Extracting, Transforming, Loading

My team of 3 and I built a microservice-based, data pipeline web service that follows the ETL (Extract, Transform, Load) architecture using Go.

 In a real ETL Pipeline, data is continuously arriving (e.g., as visitors browse a website), this implementation is a simplification where one uploads a log file to be processed once in a batch. That being said, although a simplified version of the real deal, this was a great way to become a bit more familiar with Go and the benefits of microservice in comparison to monoliths.

ETL Pipelines are a popular way to take input data (in this case web log files), extract the significant fields from the log entries, transform the data into a more useful format, and load the results into a database for future reporting.

Languages: Go