Journey from Chaos to Management — Introduction to Automation | by Greg Rog | The Startup | Apr, 2024

Right now I slept by means of a catastrophe.

At 4:35 a.m. I received the textual content message that one in every of my servers is down. Motive? Trivial. No funds on a related bank card. Did I miss some notifications beforehand? No. I flip all of them OFF by default. I consciously opt-out of any issues which will distract my work and life. And with over 150 subscription providers that my enterprise handles I’d be notified each millisecond. However hey! Shouldn’t notifications that come earlier than the blackout be AT LEAST those that I opt-in for? No.

I’ve robots to cope with them 🤖

At 4:35 one in every of them received woken up. He checked the steadiness on my completely different disposable bank cards, took the primary one with enough funds and plugged it in through internet hosting service billing API. At 4:36 I received one other textual content message. Server Standing: up.

I noticed the messages after I went biking, had a bathe and morning espresso, round 8. Good ol’ bots — I assumed.

The primary time the thought resembled me whereas studying about nomadlist ran by Pieter Ranges, who employed over 1000 bots on his web site, together with these calculating beer costs 🍺 I picked up the thought and began studying about cron, automation, AWS Lambda, internet scrapers and writing first small items of software program.

Holy sh*t, that’s superior! — I assumed.

I’ve realized to code early in my profession, however then switched to internet design, UX and biz dev. Though I might code something I didn’t discover it too thrilling. It was taking too lengthy and I form of missed the moment gratification you get from graphic and movement design.

Once I found bots and automation it immediately felt just like the lacking piece of the puzzle in my life simply received in place.

The outcomes are fast and highly effective and no or little code is required to make software program, that truly works.

As I used to be exploring extra instruments and strategies I found that nearly each daunting a part of my work might be changed with good automation. Ranging from organising my schedule and sending invoices to my accountant, to operating a complete on-line enterprise on autopilot, with bots taking good care of purchases and returns, invoicing, customer support and advertising and marketing. End result? Right now I run a number of on-line initiatives together with the one with 300k purchasers and solely few individuals on board, due to bots and automation.

The general public I speak to have come throughout the idea of automation. Nonetheless, their understanding of this subject is completely different from what I need to current, so I consider it is extremely essential for you to concentrate on how the evolution of automation seems to be from my viewpoint and the way I perceive this idea. For me, automation is using obtainable instruments to reduce human involvement in varied areas of on-line initiatives, comparable to gross sales, customer support, accounting, or advertising and marketing. I need to share my strategy and set of strategies that I take advantage of day by day to make my work less complicated and never contain me the place it isn’t mandatory.

A easy instance is dealing with returns the place we have to subject a corrected bill, obtain affirmation from the shopper, course of the return, and e-book it. Seemingly a easy operation, because it seems, in bigger corporations, it includes a number of positions. Polish airways LOT just lately processed my refund for a canceled flight for 7 months. That’s not all; altering firm knowledge on an bill within the Microsoft Azure service took nearly a 12 months and required the alternate of 151 emails. In the meantime, whereas the thread was open, one of many customer support representatives managed to get pregnant, give start, and go on maternity depart. In Vimeo, the same course of continues to be ongoing, I can’t depend the messages, however in the meanwhile, 14 individuals on the Vimeo facet are concerned, together with 4 managers, authorized division personnel, accountants, and a number of other others.

You could possibly say these are procedures of huge firms. In the meantime, in smaller corporations, although the method could also be sooner, more often than not nothing is described, and every case is taken into account individually, with the height of desires being template responses that barely pace up the work.

In the meantime, all repetitive duties might be delegated to robots and automatic, nearly completely, even when the method is advanced and requires affirmation exchanges. On one hand, this can be a big money and time saver for corporations the place hours are spent on boring and repetitive duties, and however, an enormous alternative to achieve a aggressive benefit, and even compete with a lot bigger corporations, for smaller ones. One other big advantage of automating boring, monotonous duties is unleashing the potential of people that might be rather more inventive and glad with their work.

For years, there was a perception (and few nonetheless assume in a different way) that automation requires specialised information and armies of programmers. Certainly, till now, course of optimization was primarily accomplished by bigger corporations that employed RPA (Robotic Course of Automation) specialists, individuals who first spent 3 months recognizing a course of, then produced devoted software program for a number of months, after which maintained it indefinitely. You could possibly say that automation was solely obtainable to the biggest and wealthiest.

In all probability essentially the most revolutionary change on this space is the evolution of instruments that help automation and, consequently, the democratization of those strategies. Right now, actually everybody is ready to automate, and most significantly, reaching a really superior degree doesn’t require any programming information. Moreover, instruments supporting automation are reasonably priced, which additionally ensures huge entry to expertise.

When the Web turned nearly universally accessible, choices for knowledge synchronization within the cloud and the power to collaborate concurrently on the identical knowledge emerged. Initially, there have been challenge administration purposes after which doc modifying instruments (Google Apps). Right now, even designers can work concurrently on the identical challenge, for instance, utilizing Figma. This clearly reveals a development, main us from a state of affairs the place purposes labored independently to the purpose the place they moved to the Web and browser-based variations. This facilitated collaborative work and entry to our knowledge from any gadget.

And since we’re speaking about units, we now have gone by means of a revolution associated to cell units, or reasonably, multi-platform help. This has made the method of software program improvement and design rather more advanced. Now, builders not solely needed to develop the primary model of the applying but additionally deal with their cell variations for smartphones and tablets. This led to the technique of designing utility structure utilizing APIs, which stands for Software Programming Interface.

To know what an API is, it is very important know that an “interface” is a method to allow communication. And an API is a method to allow communication inside one or a number of purposes.

This technique simplified the method of making purposes for a number of platforms as a result of builders now not needed to create a number of variations of the applying. As an alternative (utilizing a simplified schema), a single server offering consumer knowledge was enough for purposes created on particular platforms, with a decrease degree of complexity on this case.

Moreover, not solely the builders of a selected utility have entry to a well-prepared API but additionally anybody who obtains entry keys (e.g., API key).

All of this led to the creation of a software program improvement technique referred to as “API First”. From what I’ve simply written, this strategy is absolutely justified. If we will write one code to deal with a number of platforms, why wouldn’t we do it?

And what does this imply for non-programmers? Properly, the API-First strategy can be utilized as a enterprise technique and for using purposes — due to automation.

As we now have already talked about, APIs enable for knowledge alternate inside purposes and between purposes. This opens up completely new potentialities. It seems that we will join varied providers we use and alternate knowledge between them. This implies we’re now not restricted by the capabilities obtainable in chosen purposes however have a full vary of potentialities supplied by all of the instruments we use!

How is that this doable? Properly, builders present entry to their API not solely to their very own purposes (for instance, the cell and browser utility of your financial institution), but additionally… to different builders, who can construct options primarily based on their software program.

These different builders will likely be us, utilizing instruments specialised in particular duties — sending emails (e.g., Mailchimp), SMS messages (e.g., Twilio), enriching buyer knowledge (e.g., Clearbit), notifying the gross sales division (e.g., Slack), and so forth., and mixing them by means of Automation. With this strategy, we are going to by no means need to log into the graphical interface of the purposes we use, for instance, to ship advertising and marketing emails. Automation will do it for us by “logging in” to the API and sending emails of particular content material to a delegated group of recipients. Because of this neither we nor our workers will ever need to log in (and neglect passwords) to the net providers we use, which have good, accessible APIs.

The advantages of this strategy embrace:

• No want to make use of GUI (reloading pages, gradual software operation, forgetting passwords, logging in once more — all of those waste time!)
• The power to attach the very best, specialised providers as a substitute of counting on one supplier who might always lack sure performance
• Reactive work, not proactive work — we don’t need to verify what must be accomplished; automation assigns duties that people ought to full within the course of (extra on that later)

Taking a look at this by means of the lens of my earlier definition of automation, a technical side might be added to the checklist:

1. Detailed course of mapping
2. Try to simplify/enhance the method
3. Choosing the proper instruments for the fitting duties within the course of
4. Connecting these instruments by means of APIs to work on the proper time, with out the necessity for human intervention and logging into GUI
5. Supervision, error correction, and experiments

Let’s take the instance of organizing a web based occasion, comparable to a webinar. The webinar software permits for participant registration, so we now have entry to electronic mail addresses and cellphone numbers. With the assistance of API and automation, we will robotically switch this consumer’s knowledge to the Clearbit service chargeable for Knowledge Enrichment, which enriches knowledge. By sending, for instance, a company electronic mail handle, we obtain further details about the particular person signing up for our occasion. These enriched knowledge will also be robotically despatched to our CRM system (e.g., Pipedrive), positively impacting the gross sales course of as a result of when contacting such a possible buyer, we now have extra details about them to raised tailor our providers to their wants. Alternatively, we will use the cellphone quantity to robotically ship a brief SMS reminder in regards to the occasion. This will increase attendance, which might profit us and our potential prospects.

If we mix this with a system that collects buyer knowledge on our web site, comparable to Intercom, our salesperson can even obtain exact notifications with data like:

Adam from the Browser firm logged in 10 minutes in the past, he participated within the “Monetary Markets” Webinar and visited the pages: provide, faq, contact. His cellphone quantity is: 888999000

As you possibly can simply think about, utilizing one software on this course of is not possible. There isn’t any single software that may deal with webinars, web site knowledge, firm communicators, and so forth. And that is the ability of automation — we use varied instruments specialised in particular duties, juggle their knowledge, and the outcomes go precisely the place our staff feels comfy.

So, as you possibly can see, we used the API right here to switch knowledge between completely different providers and thus optimize a course of that was both beforehand accomplished manually or not accomplished in any respect. This strategy streamlines current processes or creates new ones. I’ve personally discovered that within the case of on-line companies, this strategy actually takes our initiatives not simply to the following degree however usually ten ranges increased.

For instance, the initiatives we now have undertaken, contemplating the dimensions of operations and their profitability, can simply be in comparison with companies that require the involvement of even a staff of dozens of individuals!

As we established above, trendy purposes are more and more speaking by means of APIs. Let’s take Fb for instance. The enterprise logic is chargeable for including your publish to a sure sort of database and your feed. This logic might be triggered from varied locations. You’ll be able to add a publish by means of a cell utility or the Fb web site, and as you recognize nicely, different purposes can join with Fb and publish in your behalf.

To implement this logic, it’s essential:

1. Know the place the back-end expects this knowledge (an endpoint URL)
2. Ship the information in a means comprehensible to the back-end (often in JSON format)
3. Authenticate utilizing sure entry knowledge (login, token, and so forth.)

And voila! The publish is in your feed.

Your complete web works this manner. And due to this mannequin, everybody can add, modify, and execute instructions in internet purposes.

However as a substitute of turning into a developer and studying learn how to construction, format, and ship knowledge by means of varied APIs, you should utilize instruments that can do it for you. I name them LEGO instruments. Whenever you authorize any utility in such instruments, they will carry out varied actions in your behalf. For instance, create a spreadsheet and add rows to it.

Now, let’s take into consideration this extra broadly. If LEGO instruments can join with completely different purposes, they will additionally alternate knowledge between these purposes. For example, these instruments can take rows from a spreadsheet and publish them in your Fb feed. Sequencing, timing, delays, situations — every thing might be set.

Now think about you need to collect knowledge out of your spreadsheet and distribute it to all of your social media profiles — Fb, LinkedIn, Instagram, YouTube, unexpectedly. LEGO instruments assist you to join with quite a few purposes and providers you employ and alternate data between them or work on modifications in a single software whereas notifying others.

Listed here are a number of instance eventualities:

• Looking for occasions in your calendar and sending you reminders about a very powerful ones;
• Gathering duties you enter into Trello and syncing them together with your Google calendar;
• Sending emails to webinar members, posting on social media, and assigning duties to your staff on Slack — all concurrently.

The great thing about LEGO instruments is that they broaden the capabilities of the software program you already use, permitting you to attain virtually something you possibly can think about. Usually, the software you’re utilizing might lack only one function, comparable to textual content reminders. With LEGO instruments, you possibly can add a textual content messaging service to your current arsenal of choices in just some minutes.

The most well-liked LEGO instruments are zapier and Make. Its foremost benefit is that it helps tons of of accessible purposes. It presents a transparent and intuitive interface for creating automations. Make It has a really fascinating, visible interface that lets you construct automations by actually connecting a number of dots. Moreover, it presents a wealthy, free plan to experiment with automation.

These are the instruments I take advantage of most frequently, however there are additionally many different instruments, often designed for extra particular duties.

• Parabola — an excellent software for knowledge transformation. This implies reorganizing, splitting, altering the order, formatting, and rather more. If in case you have knowledge from a number of sources or a number of knowledge that you just need to remodel and ship to different instruments, this can be a nice selection.
• Actiondesk — primarily based on a spreadsheet interface, permitting manipulation, transformation, and creating logic inside your knowledge. If you happen to like spreadsheets, it’s value trying out this software. It lets you make varied transformations on knowledge and ship it to a different software.
• Retool — this software fetches knowledge from APIs or databases and lets you remodel it and even ship instructions again for execution. If you wish to create an admin interface in your database or a report panel, this can be a good possibility.

Moreover, there are different instruments, created for particular actions, that I don’t categorize as LEGO instruments — they assist you to mix one or two providers and create one thing new. An excellent instance is Glide, which helps you to create a cell app from a spreadsheet. There are lots of fascinating instruments of this sort, however that’s a subject for an additional article.

If you wish to study extra about completely different instruments and strategies we use for automation and productiveness, be sure to try On a regular basis on Autopilot — our productiveness and automation bible, presently 50% off for Techsistence readers solely 💥 That is the very best begin to your automation journey. Right here’s few current evaluations from Product Hunt.

I believe one in every of prime books in it’s class. Reveals fascinating strategy to automation. I get to know bunch of latest instruments from it.
— Ben

Imho this book and different data merchandise from this authors are the very best quality and most advanded soruce of knowlege about proces automation and productiveness improvment! Shut up and take my cash once more! 😉
— Jakub

Thoughts-blowing information. There’s no fluff solely apply-at-once options. Very dependable. Value each minute and cash spent. I assumed I knew one thing about automation and was having fun with driving a motorcycle as a substitute of strolling… these guys confirmed me Floo Powder!
— Karolina

Get 10x extra productive! Think about the essence of our information about automation, excluding sophisticated strategies, which you’ll implement even in the midst of one night. This bundle serves because the cornerstone of our work, offering a strong basis for optimising your duties and automating day by day actions.

All of this will likely appear overwhelming. However when you perceive the idea behind LEGO instruments, it is going to be fairly simple to implement and study any new instruments. It’s value figuring out no less than a number of of them. Generally the applying you need to automate is accessible in Zapier however not in Make. Generally you need to work with a bigger quantity of knowledge, and Parabola could also be extra handy. Sadly, there isn’t a single LEGO software that meets all of your expectations, however that’s… not a foul factor. By studying completely different instruments, you encounter numerous eventualities and thought patterns.

Once I absolutely grasped how Zapier works, I used to be very excited. However then I struggled with implementing this idea into my enterprise. Tutorials have been obtainable however taught every thing from the incorrect angle, saying “join X to Y” as a substitute of “obtain X by doing Y.” This made it very tough for me to work with these instruments, and it took me over a 12 months to provide you with good automation methods and their utilization.

First automation we are going to put together primarily based on one of the well-liked platforms on the earth — Zapier. This software could be very beginner-friendly and lets you join varied instruments and switch knowledge between them with out coding. Our first automation will present you learn how to work with Zaps, as Zapier calls eventualities, or particular person “robots”.

What is going to we automate? We are going to create a easy automation that can save your Gmail attachments to a Google Drive folder. Then we are going to barely modify it to have a sensible utility by saving the invoices you obtain in your mailbox in a single place.

Begin by making a free account on Zapier. To do that, go to zapier and register, then create a brand new Zap by clicking this button, after which Open Editor, making a Customized Zap.

Now it’s essential add a set off — one thing that can begin your robotic. Seek for and choose the Gmail utility. There’s an experimental AI search above, however we’ll skip it to raised perceive the method

Within the subsequent step, it’s essential join your Google account with the Zapier utility and specify the suitable mailbox. You solely do that as soon as, and subsequent automations is not going to require reconnection

The set off occasion for us is New E mail Matching Search, which means we need to set off the robotic each time we obtain a message with particular subject and attachment

Subsequent, we will fine-tune our set off, and right here Zapier presents a really fascinating chance. We will carry out extra advanced queries, based on the Gmail search documentation. For instance, I used bill OR invoice AND filename:pdf, which suggests amongst messages with attachments, it would solely seek for these containing the phrase bill or faktura (or related as a result of I did not use citation marks). Moreover, I’m solely wanting amongst outcomes containing PDF as an attachment

You’ll be able to check your automation with the Check Set off function, and if the mailbox already accommodates parts that match the entered phrase, it’s best to obtain them as outcomes (if there’s nothing, simulate check knowledge by sending your self an bill to the mailbox)

And it’s best to be capable to see outcomes

Choose the suitable outcome and click on Proceed.

Nice! We have now configured our set off, which means we will obtain messages with attachments, moreover filtered. On this case, we solely have to configure what’s going to enable us to avoid wasting the attachment to Google Drive. As a part of the Motion (what occurs on account of the set off’s motion, we seek for Google Drive)

Then, we select what we need to do, which is to add the file to Google Drive

Join your Google Drive account, or choose it if it has been beforehand related

If you happen to haven’t created the suitable bill folder on Drive earlier than, now go to drive.google.com and create a folder named invoices. If it’s already created, it is possible for you to to specify it as the situation in Zapier

Now click on on the file and from the drop-down checklist, choose the Attachment possibility, merely passing the attachment as a file to Drive. Go away the opposite fields unchanged, apart from the File Identify subject, the place you will get inventive and alter the title of the saved file, utilizing varied metadata from the e-mail, such because the sender’s handle, combining it with the file title. I added hyphens between them for readability

Tip: If you happen to paste the next phrase within the Zapier subject: {{zap_meta_human_now}}, Zapier will substitute the present date in that place. By formatting names on this means, it is going to be simple to seek out information with the proper date. Moreover, sorting by title or date will now return neatly organized outcomes, which suggests you need not create further folders (however if you wish to do it, be at liberty to take action. Attempt modifying the automation to put invoices in a folder with the proper title, for instance, January. You’ll need to create these folders manually or with the assistance of one other automation).

That’s it! Choose Proceed, after which Check & Proceed to verify the automation’s operation. If every thing went easily, you will notice successful display

Now you possibly can Publish your state of affairs! Check bill is already on Google Drive

Bear in mind to activate the Zap utilizing the change on the backside in order that it begins analyzing all incoming messages any further. In any other case, the Zap gained’t work!

We ready our first automation in Zapier for a motive. Make, the applying we will likely be working with now, just isn’t nearly as good at integrating with Google instruments. You could possibly say that Zapier is best aligned with Google, and this can be a good instance of why it’s value having varied automation options and utilizing those that work finest in a given state of affairs. In Make, to make use of Gmail, it’s essential have a paid Google Workspace account.

Though automating with Zapier is extremely easy and intuitive, there are numerous different instruments on the earth of automation that you’ll use. Why? There are lots of causes. For instance, Zapier is a bit costly, and yow will discover cheaper alternate options. Moreover, whereas it presents many fascinating options, different instruments typically work higher and performance barely in a different way. In Zapier, though it’s doable, splitting eventualities primarily based on situations just isn’t essentially the most intuitive.

For instance, we will create a bit extra superior state of affairs in Make, that searches for invoices then renames, catalogs and save them to drive, checking if the folder construction is appropriate. It seems to be like this:

On the left, we now have the set off, which listens to emails on Gmail. Then we rename the file, and seek for the suitable folder on drive. The folder ought to include month and 12 months to catalog our invoices, so if it doesn’t exist, we are going to create it within the subsequent step after which put the bill there. You don’t want to grasp precisely what these parts imply proper now, however the idea must be a bit clearer now.

What is going to we automate this time? We gained’t repeat the identical job in Make. As an alternative, we are going to add one other automation that can ship a notification on the tenth day of the month, informing our accountant or our electronic mail handle that the collected paperwork are prepared.

Create an account on Make (that is our particular hyperlink, that grants you 1 free month on the professional plan 🎉)

Create a brand new state of affairs (equal to a Zap) utilizing the Create State of affairs possibility

Click on on the primary circle and choose the Gmail utility

Select the motion Ship an electronic mail

Click on Add within the Connection possibility, title your connection, and like in Zapier, hook up with the suitable Google account

Now add recipients, topic, and content material of the despatched electronic mail together with the hyperlink to the shared Invoices folder on Drive

Choose OK to verify the modifications within the module

Now set the schedule, for the Gmail module, click on on the clock, after which schedule sending this electronic mail on the tenth day of the month

Save the module by clicking OK

Flip scheduling to ON and also you’re accomplished!

This can be a tremendous easy state of affairs, however introduces a pleasant idea of scheduling and operating your robots remotely. If you wish to broaden it, strive sending a Slack message to your self with affirmation, after you ship electronic mail. Small tip:

You have already got your first automations in your account — congratulations! As you possibly can see, it’s baby’s play. In fact, these are simply the fundamentals, and in follow, automations might be rather more advanced, together with, for instance:

• Many extra modules and routers, i.e., paths
• Conditional directions “carry out solely when the information accommodates X”
• Capabilities, for instance, looking out by means of returned knowledge
• Knowledge completion, for instance, with the present date or knowledge calculation utilizing mathematical operations
• Error dealing with and modules for testing
• Customized scripts, for instance, JavaScript
• Modules permitting knowledge reformatting (parsing)
• Modules able to iterating, for instance, over information in a database or rows in Google Sheets
• And so forth

There are many potentialities. However let’s break down what actually occurs within the eventualities we created. The truth that Zapier or Make permits us to decide on the Gmail module merely implies that the software can combine with it. How? In fact, by means of APIs! For instance, Gmail has an API, the place we will check with the suitable endpoint and hearken to incoming emails. As soon as we retrieve details about the incoming message, the following endpoint permits us to obtain the attachments. Zapier, which we use, simply has this stuff built-in, however they work on the usual Gmail or Slack API. We work precisely like a programmer who would put together an integration for us, besides we use a visible builder and a set of ready modules. The account connection we made is actually the authentication within the API utilizing one of many strategies mentioned earlier. It’s programming with out programming!

Make, like Zapier, presents the likelihood to make use of tons of of ready-made eventualities or import any blueprints ready by us or others!

As you in all probability guess, through the years we’ve created various automation for our wants. You is likely to be shocked, however presently over 3000 robots work for me, performing tons of of 1000’s of operations day by day. That is simply one of many Make accounts I take advantage of, the place I course of almost half one million operations month-to-month!

However that is only the start! Slightly below a 12 months in the past, we realized how helpful NLP and AI might be to assist us work with automations. It turned out that enter knowledge for automations, which wanted to be formatted in a selected means (we ship them the so-called JSON object), might be ready by AI! What does this imply in follow? It means which you can say or write, for instance:

Add shopper John Doe to CRM

After which the language mannequin will deal with translating this into the correct format, which means into the “automation language,” after which ship it to a prepared state of affairs on Zapier or Make! It’s actually wonderful how huge the chances are when utilizing such an strategy.

Sadly, there wasn’t a software that allowed us to speak AI with automations on this means, so over a 12 months in the past, we began engaged on our personal resolution to assist us with this.

And that’s how Alice app was born.

Along with with the ability to chat with Alice like with ChatGPT, her greatest benefit lies within the potential to remotely set off our automations, and consequently — carry out actions! ChatGPT presents this idea additionally with GPTs and their API commonplace, nonetheless, this strategy proved to be a lot much less environment friendly thus far, no less than for our use-cases.

Way forward for distant actions in Alice could be very promising, and possibly we are going to, in time, begin including native integrations to Alice. Additionally, efficiency of LLMs is quickly growing, and likewise entry to native fashions in Alice is a chance. Right now with Groq we’re capable of carry out operations sooner than ever earlier than, and it will evolve. Check out this command to Spotify and the pace of its execution.

On this context, I’ve ready some tutorials for you, exhibiting how one can combine the Alice app with Make or Zapier. Please, see the way it works in follow!