(You may not be familiar with what elasticsearch is : for a good introduction checkout this post from Igor Kopanev)
Extracting data from ElasticSearch in golang is not a simple task. First of all you might not be the one who wrote data into it so these are some commands to find out how is data structured in your ES server :
List all your indexes
$ curl https://<your-server>/_aliases?pretty=true
{
"billingdata" : {
"aliases" : { }
},
"checkpoints" : {
"aliases" : { }
}
}
Your indexes data structures can be queried with :
curl https://<your-server>/<index_name>
Now that I had an idea of how data was structured I started looking at the ES official library ( https://github.com/elastic/go-elasticsearch/ ) to find out that it is probably not used that much directly. This a code example to execute a query :
// Build the request body.
var buf bytes.Buffer
query := map[string]interface{}{
"query": map[string]interface{}{
"match": map[string]interface{}{
"title": "test",
},
},
}
if err := json.NewEncoder(&buf).Encode(query); err != nil {
log.Fatalf("Error encoding query: %s", err)
}
// Perform the search request.
res, err = es.Search(
es.Search.WithContext(context.Background()),
es.Search.WithIndex("billingdata"),
es.Search.WithBody(&buf),
es.Search.WithTrackTotalHits(true),
es.Search.WithPretty(),
)
Then (God bless Oliver Eilhard) I ray of light came in through my window and I found this package : https://godoc.org/github.com/olivere/elastic that makes everything much more simple :
searchResult, err := es6.Search().
Index("billingdata").
// Query(timerangeQuery). // if you need to filter
Pretty(true).
Do(context.Background()) // execute
Much better. If you need a time range query :
// sdate,edate are time.Time
timerangeQuery := elastic.NewBoolQuery().
Filter(elastic.NewRangeQuery("@timestamp").
From(sdate).
To(edate))
Or maybe you need to query by time range and field value :
companyQuery = elastic.NewTermQuery("company_id", CompanyId)
// combine queries :
CombinedQuery = elastic.NewBoolQuery()
CombinedQuery = CombinedQuery.Must(timerangeQuery).Must(companyQuery)
// now search using Query(CombinedQuery)
Setting up your ES Client is pretty easy :
// Create a custom HTTP client to setup timeouts and TLS config
// esHTTPClient := &http.Client{
// Timeout: time.Second * 10,
// Transport: &http.Transport{
// Dial: (&net.Dialer{
// Timeout: 5 * time.Second,
// }).Dial,
// TLSHandshakeTimeout: 5 * time.Second,
// TLSClientConfig: &tls.Config{
// InsecureSkipVerify: true,
// },
// },
// }
// Create a client
client, err := elastic.NewClient(
// elastic.SetHttpClient(esHTTPClient),
elastic.SetSniff(false),
elastic.SetURL("http://localhost:9200"))
You can also aggregate data easily, sum, avergage. I’ll come up with more examples as soon as I have some working code.