I am a Software Engineer at heart. I started as such and worked with PHP for about 7 years, always correlating my work with data somehow until I got an opportunity and decided to follow my instincts and be a Data Engineer.

I didn’t turn a Data Engineer from one night to another. It was a process. I was lucky to have a boss that noticed my skills with data and decided to give me room to play with it.

But the Ops part, was never my forte and this is why.

Data Integrity

This is my main concern. I am more worried about keeping consistency as much as possible and even in many times choosing it over performance.

Another trait I have is to be always looking for logic errors that may generate bad data into an application. I despise badly written models and have had a bit of problem when working with RDBMS and ActiveRecord. My take on it is: if you have a complex business model, what is easy may become painful. Also, there is no silver bullet solution, you don’t need to use only one technology.

But I won’t go into what is better, what is better it is what it works for you and make you application works and don’t let your users down while maintaining your data integrity.

Value

Your software is not valuable 99% of the times. Your software it is a means to an end. It is a path to interpret business logic and generate value to your company. And if your data sucks (duplicated records, lack of foreign key checks, extreme denormalization in the main DB if relational) you may have no real value at all.

Why don’t I know it?

When I needed doing ops, managing database servers, they were all single servers, at most a read replica on AWS, or only one slave.

One can say I’ve never needed actually to know it. I got lucky having good people working with me on the DevOps team, and we trusted each other’s work, in the end the managers would prefer to use my abilities in another area.

As I said before I do know it’s an area I need to improve, but it is ok to not know it, because even with me not being an expert on it my value lies in understanding the data, the data model and how application handles on data. As a DBA main job usually is to keep the database servers healthy, mine is to keep data itself as a valuable as possible to a company.

Not a DBA

What I do it is many times considered a DBA job, there are a couple areas where both can overlap but I try as a Data Engineer to support the Developers and the Business as a DevOps person do.

You may notice on my posts: I don’t write about replication, cluster, etc. One reason is: I have never in real life have to deal with this particular area deeply. I do know I should know more about that. But one thing is to set it up a couple servers on the cloud with no real data to analyze and performance issues to attend than doing it in real life. However, that doesn’t lessen my value.

I know how to prepare data, I do ETL’s, I do data modeling, I do deep research on which storage would be the best for a case scenario, I help to define policies around migrations and data access, for instance.

My Goal

My goal is to help developers. It’s to help them do the right thing regarding to data as they do regarding with test coverage and code quality.

Everybody reviews code, I rarely seem people reviewing data models. And I want to help to create a culture where people see data as their true value. Remember this: data leaks are more valuable than “code” leaks and potentially more devastating too. So please, let me help you.

• Article