我们都知道在分布式日志当中,exceptionless客户端是把日志写到
Elasticsearch数据库,就像我们把数据写入到关系数据库一样;
既然是写入,那么在短时间大数据量的情况下,写入就会涉及到效率的问题;.首先我们看下 exceptionless源码
public class InMemoryObjectStorage : IObjectStorage {
private readonly Dictionary<string, Tuple<ObjectInfo, object>> _storage = new Dictionary<string, Tuple<ObjectInfo, object>>(StringComparer.OrdinalIgnoreCase);
private readonly object _lock = new object();
public InMemoryObjectStorage() : this(1000) {}
public InMemoryObjectStorage(int maxObjects) {
MaxObjects = maxObjects;
}
public long MaxObjects { get; set; }
public int Count {
get { return _storage.Count; }
}
我们写入到日志的方法 ,首先会把日志写到内存当中;但是为了防止日志过大,导致服务器的内存被过度消耗的问题 ,这边是限制了 日志的条数的 1000;
当有新的日志进来时,如果是超过1000,那么旧的日志是会被抛弃的;
public bool SaveObject<T>(string path, T value) where T : class {
if (String.IsNullOrWhiteSpace(path))
throw new ArgumentNullException("path");
lock (_lock) {
_storage[path] = Tuple.Create(new ObjectInfo {
Created = DateTime.Now,
Modified = DateTime.Now,
Path = path
}, (object)value);
if (_storage.Count > MaxObjects)
_storage.Remove(_storage.OrderByDescending(kvp => kvp.Value.Item1.Created).First().Key);
}
return true;
}
这显然不是 我们想要的,我们需要的是完整的日志;
public SubmissionResponse PostEvents(IEnumerable<Event> events, ExceptionlessConfiguration config, IJsonSerializer serializer) {
if (!config.IsValid)
return new SubmissionResponse(500, message: "Invalid client configuration settings");
string data = serializer.Serialize(events);
string url = String.Format("{0}/events", GetServiceEndPoint(config));
HttpResponseMessage response;
try {
HttpContent content = new StringContent(data, Encoding.UTF8, "application/json");
// don't compress data smaller than 4kb
if (data.Length > 1024 * 4)
content = new GzipContent(content);
_client.Value.AddAuthorizationHeader(config.ApiKey);
response = _client.Value.PostAsync(url, content).ConfigureAwait(false).GetAwaiter().GetResult();
} catch (Exception ex) {
return new SubmissionResponse(500, exception: ex);
}
int settingsVersion;
if (Int32.TryParse(GetSettingsVersionHeader(response.Headers), out settingsVersion))
SettingsManager.CheckVersion(settingsVersion, config);
return new SubmissionResponse((int)response.StatusCode, GetResponseMessage(response));
}
所以我们需要在PostEvents 这个方法做文章;
目前我们的做法是:
就是把日志直接写到消息队列,然后做一个消费日志的订阅队列的服务,从队列读取数据,在写到Elasticsearch。