Coralogix
Stateful Streaming Analytics for Observability Data
Real-time insights and trend analysis for logs, metrics, and security data with no reliance on storage or indexing
We use Coralogix to collect logs from all our environments from QA to PROD.
Each environment has its own account in Coralogix and thus its own limit. Coralogix price module are calculated per account.
We as a company have our budget per account and we know how much we pay per each one.
In case you exceed the number of logs assigned per account you will pay for the “extra” logs. You can see the exact calculation in this link
Our Flow:
You can see our flow below showing each ENV disconnected from the other but all of them under our account in Coralogix.
The problem
In each environments (Except PROD) we allow our developers to decide what will be the log level they want to write, and it can cause somewhat of an issue if you constantly writing in DEBUG or VERBOSE. You can reach your Coralogix quota quite fast if you are not careful.
We needed a way ( without chasing the developer teams each day with a slap on the wrist ) to limit the amount of logs with no human interactions.
The solution
We had a few options to consider on how to do it:
- Availability ( 24/7/365)
- Not environment dependent
- Ability to access Coralogix API
We wanted a solution that will not be part of our stack and will always run against Coralogix API.
We chose to use Azure Functions
Azure Functions
Accelerate and simplify serverless application development with serverless compute
Azure function along with AWS Lambda and Google Cloud Functions were our main focus and we chose Azure function as we are already working with Azure and they provide 1 million executions on a free tier so the choice was easy.
The functions were written in python and you can see the flow below:
Coralogix Rules
As you might have seen in the above diagram we use Coralogix rules to stop logs from being parsed and thus save money on ingested logs every single day. What are rules:
Rules help you to process, parse, and restructure log data to prepare for monitoring and analysis
Coralogix offer many different types of log parsing rules like:
- Parse
- Extract
- Extract JSON
- Replace
- Block
- Timestamp Extract
- Remove Fields
You can see the full list at the Coralogix site:
https://coralogix.com/tutorials/log-parsing-rules/
In our case we used the Block option. Block rules allow you to filter out your incoming logs using RegEx.
The rules are part of log groups that can contain multiple rules.
See example here:
Exclusion list
Per request from our developers we added a way to unblock an application for a predefined period of time OR give them an added XXX lines of logs to be parsed and displayed in the UI before they are blocked again:
Conclusion
We had a necessity to lower our log collector SaaS cost and using azure functions we were able to moderate it to a manageable flow.
Most important we now have visibility to what application is costing us the most and we can work closely with the Dev team to reduce the amount of logs they write.