Skip to main content

· 8 min read
Mohammad Teimori Pabandi

If you have ever tried to deploy your containerized application to ECS, you know that it's not going to be an easy click-to-deploy journey. In order to get your application up and running on ECS, you have to go through a bunch of resource creation. You'll need to:

  • Deploy a load balancer as the point-of-contact for your app
  • Create a target group for the load balancer and register the ECS tasks in it
  • Add a new listener to your load balancer and connect it to the target group
  • Create a security group, and you need to allow the port your app is listening on in that security group
  • Attach the security group above to your load balancer
  • Create a CloudWatch log group for your ECS task
  • Create an ECS cluster, and definitely the task definition as well
  • Oh, and create an ECR repository to push your images to be run on the container

I'm not going to continue this long list, since I've already got a headache. Doing those steps manually is going to give you a headache as well, so why bother doing all those steps yourself and risking different errors you might face when deploying your containerized app? You don't really have the time for the random IAM-related errors AWS is demanding you to resolve. Besides, you already have your codebase ready and the Dockerfile is there, so why not just run a simple command doing something that should be simply done?

In this post, we're going to discover an IaSQL module that's built to make deploying to ECS, simplified. Most of the details for deploying a container to ECS are the same (load balancers, security groups, IAM roles, etc), and we have created the aws_ecs_simplified module for you so that you can give it any Github repo with a Dockerfile and get your app deployed to ECS in the fastest time possible, with scalability available! All the needed resources are going to be created automatically in your AWS account, and you'll have the full access to the details while you're gaining the benefit of a higher-level simple deployment.

An Example Usage of the aws_ecs_simplified Module

Let's say we are going to deploy this simple Express.js app to the ECS. It has a Dockerfile and package.json that installs express on npm install. npm start then starts the Express server which listens on port 8088.

note

aws_ecs_simplified is a high-level module we have created to make scalable ECS deployments easier. For more info on high-level vs low-level modules, you can check this guide.

Let's go and deploy the above app to your AWS account. Don't worry if you don't have an IaSQL database already, you can create one for free when you click the "Deploy Now" button.

Deploy a simple Express.js app from a Github repository to ECS
SELECT iasql_install('aws_ecs_simplified');
INSERT INTO ecs_simplified(app_name, app_port, image_tag, public_ip) VALUES(
'simple-express',
8088,
'latest',
true
);
SELECT iasql_apply();

SELECT ecr_build(
'https://github.com/iasql/iasql-engine/', -- the Github repo URL
(SELECT id
FROM repository
WHERE repository_name = 'simple-express-repository')::varchar(255), -- ECR repo for the image to be pushed
'./examples/ecs-fargate/prisma/app', -- the subdirectory in Github repo
'main', -- the Github branch or ref
NULL -- Github personal access token - can be omitted if public repository
);

That's it! Now wait for some time and your app is deployed! While your app is being deployed, let's go through the commands we executed in more depth:

SELECT iasql_install('aws_ecs_simplified');
  • This command installs the aws_ecs_simplified high-level module. We– at IaSQL– have created that module to make it easy to deploy containerized apps to ECS. The code for it is here. But IaSQL is so flexible that anyone can create their own high-level (and of course, low-level) modules and add it to IaSQL.
INSERT INTO ecs_simplified(app_name, app_port, image_tag, public_ip) VALUES(
'simple-express',
8088,
'latest',
true
);
  • This command creates a new ecs_simplified app by inserting a new row into the ecs_simplified table. Seems pretty easy, right? But under the hood, it's creating all the necessary resources like load balancers, security groups, IAM roles, etc.
  • You can manually check the tables to see what resources are being created. For example, looking at the load_balancer table you'll see a load balancer named simple-express-load-balancer is inserted automatically by running the above insert command.
SELECT iasql_apply();
  • The insert command did create the list for what resources should be created, but they've not yet created on the AWS. You need to call the iasql_apply() function to sync the cloud with the data on your database.
  • For more info on the iasql_apply() command, check this guide on how it works.
  • After running the iasql_apply() function to completion all the necessary resources are now created on the cloud, so their ARNs are populated. You can verify this by looking at different tables, eg. iam_role.
  • To get your load balancer address, you can easily run SELECT load_balancer_dns FROM ecs_simplified WHERE app_name = 'simple-express' query and get the URL to access your app.
  • Now ECS is waiting for an image to be pushed to your ECR repository to run it. You can get the URI for the ECR repository by running the SELECT repository_uri FROM ecs_simplified WHERE app_name = 'simple-express' query. You could build your docker image locally and then follow Steps 2 and 4 from this guide to connect your local docker CLI to your ECR repository and push that docker image into your ECR repository, but we have a simpler solution next.
  • In the next step, we'll automatically build an image for the code in Github repo and then push it to this URI (all through SQL and using another high-level function named ecr_build).
SELECT ecr_build(
'https://github.com/iasql/iasql-engine/', -- the Github repo URL
(SELECT id
FROM repository
WHERE repository_name = 'simple-express-repository')::varchar(255), -- ECR repo for the image to be pushed
'./examples/ecs-fargate/prisma/app', -- the subdirectory in Github repo
'main', -- the Github branch or ref
NULL -- Github personal access token - can be omitted if public repository
);
  • This command tells IaSQL to go clone iasql-engine repository, build an image on the subdirectory specified, and then push it to the ECR repository created earlier by the aws_ecs_simplified module. Running the above command will automatically create a CodeBuild project and the related roles, etc. Then it'll start a build, and after it's successful all the created resources are deleted to ensure there won't be any additional charges to your AWS account.
  • To access your app on the cloud, get the load balancer address and use your browser to access the live version of it:
SELECT load_balancer_dns
FROM ecs_simplified
WHERE app_name = 'simple-express';

Then you can check if the server is running on the <load_balancer_dns value>:8088/health address.

Low-level Access to Resources

So the aws_ecs_simplified module simplifies things, right? But what if you still need the level of control you had when you were doing all the steps manually? The traditional PaaS trade-off is that you can't grow your app beyond the built-in limitations as you don't have access to all the small details. The IaSQL approach is not limited in that way.

Let's say you want your ECS container to be able to use the AWS CLI to provision an EC2 instance, and for that purpose its IAM role needs AmazonEC2FullAccess policy to work properly. aws_ecs_simplified does not have a column to configure such a thing, but that doesn't mean we're stuck.

The good news is that you still have the full control over all resources in the deepest details. Let's fix your app's IAM role access by attaching the needed policy to its IAM role:

UPDATE iam_role
SET attached_policies_arns = attached_policies_arns ||
'arn:aws:iam::aws:policy/AmazonEC2FullAccess' -- attached_policies_arns is of text[] type
WHERE role_name = 'simple-express-ecs-task-exec-role';

You want additional rules for the container's security group? No problem! Just write the SQL and execute it, and it will be applied to the cloud within seconds. You want 3 copies of your container to be kept running with a round-robin load balancing on them? It's already there, just do an UPDATE ecs_simplified SET desired_count = 3 WHERE app_name = 'simple_express'; and it's there for you.

With IaSQL and its flexibility, you can benefit from both the high-level and low-level operations. We have created the aws_ecs_simplified module to show the flexibility and power of IaSQL engine, but the possibilities are endless. IaSQL is also an open-source project, meaning that you can use the powerful IaSQL engine and build your own modules on top of that. If you're into the idea of empowering other developers to do complex infrastructure tasks simply, why don't you take a look at our contributing guide and join our Discord channel? We'll thoroughly answer any of your question regarding the usage or development of IaSQL. Looking forward to seeing you in our small, but great community.

· 2 min read
Yolanda Robla

One of the main complexities of working with AWS is the amount of resources that can be created, either by manual or automated workflows. The chained resources and their dependencies can quickly get out of control, and it could be a nightmare to clean up then.

This is a common problem when using tests accounts, either for developing or testing some integrations. Often these accounts have resources out of control, causing you to incur on extra costs, or causing unexpected conflicts. Cleaning those manually can be a hard task... manually visiting each resource, investigating the dependencies, manually cleaning... it is a time-consuming, error-prone task. If we consider that those resources may be created in multiple regions, the manual procedure can become a very difficult task.

To solve this problem you can use our IaSQL engine. By simply clicking a button, our engine will get an snapshot of the status of your cloud, and will schedule deletion of the existing resources and its dependencies until your account is completely cleaned.

Want to try? Connect your AWS account to IaSQL for free using the dashboard. Read more about AWS account management in this part of our docs.

info

Please note that our engine can offer this cleanup only for the resources covered by our system. You can get a full coverage list here.

-- Installing all IaSQL modules
SELECT iasql_install(variadic array(select module_name from iasql_modules_list()));
-- Deleting all resources from the db
SELECT * FROM delete_all_records();
-- Previewing changes in your account
SELECT * FROM iasql_preview_apply();
-- Applying delete changes, please uncomment it to perform the cleanup
--SELECT * FROM iasql_apply();

After the initial module install, IaSQL will inspect your cloud and propose the resources to delete. If you agree with the proposed changes, please uncomment the apply statement and execute it, to let IaSQL completely cleans up your account.

· 2 min read
Luis Fernando De Pombo
David Ellis
Alejandro Guillen

We are excited to announce that IaSQL is now open source! The main repository is under https://github.com/iasql/iasql-engine. As perfectionists, we feel like IaSQL will never be truly ready. However, we believe IaSQL is at the point where it can start to be useful for developers managing infrastructure in the cloud. IaSQL is a SaaS that lets you model your infrastructure as data by maintaining a 2-way connection between your AWS account and a Postgres SQL database to represent the definitive state (and status) of your cloud which cannot be achieved with a static infrastructure declaration. This means that when you connect an AWS account to an IaSQL instance it automatically backfills the database with your existing cloud resources. No need to painstakingly redefine or reconcile your existing infrastructure, and IaSQL's module system means you can specify which parts of your cloud infrastructure you wish to control.

DashboardDashboard

IaSQL also makes it possible to express infrastructure changes as code that can be version controlled. This can be done with any migration system for schema and data changes, or in a script using idempotent SQL inserts more akin to IaC.

PostgreSQL IaSQL databases can be provisioned and configured via our dashboard. The dashboard calls the engine which is a Node.js server written in Typescript that provisions unmodified PostgreSQL instances loaded with tables representing AWS services controlled via the AWS SDK. AWS is our main focus at the moment, but we plan to support GCP, Azure and other cloud providers soon. This is an updated list of the AWS services that we currently support. Let us know if you need a specific AWS service and we should be able prioritize it!

We want to make it easier to write IaSQL modules, reduce waiting times when provisioning infrastructure, add more functionality to the existing AWS services, and so on. The list of things we want to build into IaSQL is long, but we want to do it in the open with your feedback and help. Drop us a line on Discord!

· 4 min read
Luis Fernando De Pombo
David Ellis
Alejandro Guillen

What software you have deployed on what services and the interactions between them and the outside world is not a program, it is information about your infrastructure. Changing your infrastructure is a set of operations to perform, a program. A SQL database is a set of information and SQL queries read or change that data.

Infrastructure State is Data, Infrastructure Change is Code. It's as simple as that.

And manipulating your infrastructure in this way is natural.

INSERT INTO aws_ec2 (ami_id, ec2_instance_type_id)
SELECT ami.id, ait.id
FROM ec2_instance_type as ait, (
SELECT id
FROM amis
WHERE image_name LIKE 'amzn-ami-hvm-%'ORDER BY creation_date DESC
LIMIT 1
) as ami
WHERE ait.instance_name = 't2.micro';

Relations and Types Matter for Infrastructure

Infrastructure as Code solutions do not have a good way of encoding dependencies across infrastructure pieces in a micro services architecture which makes it really hard to make and revert changes to infrastructure.

Representing your infrastructure as SQL resolves the primary issue of YAML-based infrastructure tools by making the relations between pieces of your infrastructure first-class citizens, and enforcing type safety on the data and changes to it.

You can't set the EC2 instance type as t2.mucro and have your deploy system try and fail to create such an instance. The insert statement will fail and tell you zero rows were inserted and you can quickly see why.

Similarly, if you have a record in the security_group table, you can't delete it if there are any references to it in the ec2_security_groups join table. The relational structure of IaSQL prevents you from putting your infrastructure into an invalid state.

New Powers: Explore, Query, and Automate Your Infrastructure

Because your infrastructure is presented as a SQL database, you can connect to it with a SQL client of your choice and explore what you have and what the possibilities are.

SHOW tables;

You can query for unusual usage patterns.

SELECT aws_ec2.*
FROM aws_ec2
INNER JOIN ec2_instance_type AS ait ON ait.id = aws_ec2.ec2_instance_type_id
WHERE ait.vcpus > 8
ORDER BY ait.vcpus DESC

And since it is a database, you can create your own tables with their own meaning and associate them with your infrastructure.

SELECT aws_ec2.*
FROM aws_ec2
INNER JOIN company_team_ec2s AS cte ON cte.aws_ec2_id = aws_ec2.id
INNER JOIN company_teams AS ct ON ct.id = cte.company_team_id
WHERE ct.name = 'Data Engineering'

Finally, your applications can know much more about what infrastructure they need than any auto-scaler solution out there. If you had a very infrequent but CPU/GPU-intensive job you need to handle at an unknown interval, you could give your application access to your IaSQL database and let it temporarily create and then destroy those resources.

const ec2_instance_id = await iasql(`
INSERT INTO aws_ec2 (ami_id, ec2_instance_type_id)
SELECT ami.id, ait.id
FROM ec2_instance_type as ait, (
SELECT id
FROM amis
WHERE image_name = 'application-job-runner'
) as ami
WHERE ait.instance_name = 'g3.4xlarge'
RETURNING id;
`);
await iasql(`
INSERT INTO ec2_security_groups (ec2_id, security_group_id)
SELECT ${ec2_instance_id}, sg.id
FROM security_groups AS sg
WHERE sg.name = 'application-job-group';
`);
// Only large-enough job runners will take it based on job metadata
const result = await job.run(myMassiveJob);
await iasql(`
DELETE FROM aws_ec2
WHERE id = ${ec2_instance_id};
`);

You Don't Need to Learn a New API (Probably)

Nearly all cloud backend systems depend on a database, and most likely a SQL database, so you do not need to learn a new language to manipulate the infrastructure in this way.

And likely you're using a migration system in your backend to review changes to your database, which you can continue to use here, making it code to be reviewed, just like Infrastructure-as-Code.

You Can Test, Too

Since the safety guarantees are provided by the types and relations between tables, you can simply copy your production infrastructure database into a local database and run your changes/migration against that and verify it works before you run it on your actual Infrastructure-as-SQL database.

Recover With Ease

It's 3AM and your service has gone down. You reverted the most recent IaSQL migration, but that didn't resolve the issue, and you aren't sure which change across which service today caused the outage. So, you simply replace the state of the IaSQL database with a snapshot from yesterday to bring everything back online to a known-good-state, and then take your time after you're well-rested to figure out what actually went wrong.