A few weeks ago, one of our employees at Certora fell victim to a minor hack that was quickly addressed, and that incident prompted me to reflect on my first eight months in Web3 and what I’ve learned about the differences between security in the DeFi space versus the wider tech industry. 

On the one hand, it is incredibly exciting for a cybersecurity expert like myself to join an industry that is literally on the bleeding edge of applied cryptography. It’s inspiring to see the level of sophistication in our industry and how advanced cryptography is being used to guarantee safety in a trustless system. At the same time, many of the best in our industry have fallen prey to the simplest attacks: phishing efforts that exploit vulnerabilities in communication tools like Telegram, Google Workspace, X. In the best case, falling prey to phishing is embarrassing, and in the worst case it’s very costly. 

To avoid these failures, there are several essential OpSec lessons that are well known and widely followed outside of the Web3 world that we would be wise to learn from.

Like many companies in Web3, Certora is on a long OpSec journey. When I joined the company in June, multi-factor authentication (MFA) was the only security measure enforced across the company. As many hacks in the industry have illustrated, MFA is necessary but it is by no means sufficient on its own.

From that starting point, which I suspect we share with many companies in the space, we have been progressively improving our OpSec. Our experience only emphasizes the urgency of our security roadmap. We see a constant, daily flood of phishing attempts directed at our company through all available communication channels, and both vigilance and a strong OpSec discipline are essential. In the hope of guiding others to take immediate action, I thought it might help to walk through my OpSec Top 5. We are still working through these steps while we help more and more of our clients do the same. I would recommend this same roadmap to anyone starting from a similar position.

Step 1: MFA and Sign-in Security

As noted above, MFA is necessary but not sufficient as an OpSec strategy. That said, if you have not yet implemented MFA you are making a grave mistake. Changing course had better be the first thing on your agenda as soon as you stop reading this article.

Not all MFA is created equally. I recommend the following:

1. Stay away from text and email as MFA methods. There are innumerable reasons why neither of these methods is a good idea. Suffice it to say, best practices have long since outlawed these MFA methods.
2. TOTP (e.g., Google Authenticator) is good but not great. Why? It is easy enough to trick users into entering TOTP codes into a phishing site. The methods cited below are more difficult to exploit.
3. Push-based MFA is better. Why? Because initiating a push notification on iOS/Android requires that the device itself be enrolled with the identity provider. Phishing sites cannot initiate a push notification to the Gmail app, for example, without a major compromise of Google’s infrastructure.
4. Passkeys are the best. Biometrics are hard to fake, and in a world where attackers are looking for low hanging fruit, passkeys protected by biometric factors are typically too hard for them to reach.
5. Key admins (e.g., your G suite admin) should be using Yubikeys. They are inexpensive and easy. There is no excuse here for not protecting the keys to the castle with the industry gold standard for MFA.

Once you have MFA in place, you are ready to move on to the next step. However, before you declare your MFA journey a success, make sure you haven’t forgotten any of your communication tools along the way. In this industry we often use a combination of X, Signal, and Telegram, and each of them can and should be protected with an additional authentication factor.

Step 2: EDR is a Must

As noted above, we use SentinelOne at Certora. One of our clients uses Sophos, and we both purchased and configured that Sophos system on behalf of our client. Others use Cynet, Crowdstrike, Palo Alto, or other alternatives on the market. While the MITRE attack evaluations are the gold-standard of EDR testing, it is far more important to have something in place and configured correctly than it is to have the best possible EDR according to MITRE.

One common gap in EDR implementations is that EDRs require special permissions on Mac computers. MacOS requires that a security extension is granted permission by an admin user in order to effectively protect against malware. Disk scanning requires further permissions, which must again be granted explicitly by the admin user. For those EDRs that protect against malicious browsing, there is a network extension that must be granted appropriate permissions as well. The installation process cannot force these permissions changes - they must be done manually by each user - and this can be quite a challenge with users who are less technical or may not be comfortable in the depths of the MacOS Settings app. However, without these permissions, your EDR is essentially useless on Mac devices.

It is critically important that across all platforms you use (Windows, Mac, Linux) your EDR is properly configured, has all required permissions, and you use the server-side interface to verify that all endpoints have the requisite green checkmark (or similar) indicating that protection is in effect. Don’t learn this lesson the hard way - make the time to work patiently with each user to get EDR setup and configured in the right way, then double check your work on the server side.

Step 3: Passwords Must be Managed

Adding an enterprise password manager is not a nice-to-have. It is an absolute must. Why? 

First, the best practice in password security is to have your users remember a single, complex password and to never use that one password anywhere other than as the “master” password for their password manager. It is reasonable to ask users to remember one password that is never stored online (an offline backup can be in a safe deposit box or similar). But it is not reasonable to ask users to remember more than one such password. 

If you choose a well known enterprise password manager with a strong security track record, you can be confident that they follow industry leading practices for protecting your password. Other sites may not be as careful. Use your good, memorized password as your master password, and do not reuse this password anywhere else. Ever. Also, do not store it in Google’s password manager in Chrome (or any browser). This password lives in your head and in an offline, locked, safe location.

Second, password managers insulate your users from poor password handling by 3rd party sites and they isolate the impact of a phishing attack. Users cannot remember an infinite number of complex passwords. The inevitable result is passwords that are re-used, either identically or close to it. If one such password is stolen, a smart attacker has what they need to break into every 3rd party site you use if you have reused that password. MFA helps here, but as noted above phishing attacks are getting better at coercing users into MFA mistakes. Secure, randomized passwords managed by a password manager ensure that a single successful phish has a single, localized impact.

Third, password managers allow for controlled, permissioned password sharing. Password sharing is NEVER a good idea for “hot” accounts, meaning the accounts that you use for daily work. However, many 3rd party SaaS applications have a root account. The root account should always be a “cold” account, but it still has credentials that have to be recorded and shared with a select group of company admins. Password managers are great for this purpose: they offer convenient, secure sharing capabilities. At Certora, we adopted 1Password. Some of our clients use Bitwarden. We have experimented with NordPass. All have their tradeoffs, but if you ask me to make the call, I think that 1Password is the most comprehensive solution on the market.

Once you have your password manager in place, take the extra step of resetting all your passwords to a secure, randomized, and managed password.

Step 4: Admin Accounts are for Administration

IT departments the world over have long known that you should never work out of an admin account. Email, web browsing, and pretty much anything else that you do during daily work does not require an account with permissions to install/uninstall software or to make system-wide configuration changes. When you work out of an admin account, you are one click away from a major disaster. If you work out of a regular user account, you at least have a second chance to reconsider when a password prompt appears after you click where you shouldn’t have.

Unfortunately, most users still do work out of admin accounts when using personally purchased and configured devices. It is easier to set up your system that way, but it is a terrible, awful, and dangerous practice. It is essential that all users create a standard user account with a unique password and migrate to that account for daily work ASAP. Allow yourself and your employees to make a few mistakes without triggering the next disaster. 

Once you have made this change, educate all your employees that an admin password prompt is a red flag in almost all circumstances. Unless explicitly guided otherwise during software installation or maintenance, stop at the password prompt, cancel, and ask for a second opinion before proceeding.

Step 5: Keep your Root Accounts Cold

An increasingly common and dangerous attack vector is to use OAuth/Open ID to allow a malicious 3rd party application to access a company account. Most services (e.g., X) only allow OAuth connections to be initiated from a root account. The issue, though, is that users often work out of a root account. If you have an X window open in your browser logged in via the root account to your company X account, a malicious 3rd party application is now one errant click away from an X account takeover. The same applies to many other 3rd party services. Never work out of root accounts. Instead, follow the principle of least privilege.

When you set up a new service (X, Cloudflare, AWS, etc.), create the account using a root account that you will keep cold. Nobody ever logs into this cold account unless absolutely necessary to perform a function that cannot be performed any other way. Otherwise, the root account credentials (and a TOTP for MFA) are stored in the password manager and shared with a limited group of company-wide admins who are similarly educated that this account is a cold account.

Individual accounts are created with the privileges that each user requires to accomplish their daily work. X has delegate access. Cloudflare allows for an unlimited number of user accounts whose permissions can be set at the domain level. AWS has IAM Identity Center for exactly this purpose. With few exceptions, almost every business critical application that you use can be configured in this way. If you have not done so, you are asking for trouble.

What to do when things go wrong

While you are on your top 5 journey, and perhaps even after you have completed all the steps, things can still go wrong. Phishing attacks are increasingly sophisticated, and they can fool both your users and your EDR. At some point in time, something will go wrong. Here is what you do:

1. If an endpoint is compromised, step 1 is to take that endpoint offline.
2. Once the device is offline, there is no path to recovery that is safe aside from a full factory reset and reformat. I cannot emphasize enough how important it is to adopt Google Drive or a similar online storage option so that you do not lose critical business data from a factory reset (however, never, ever set up the folder sync capability like Google Drive for Desktop … this feature is a great way for malicious software to spread).
3. If you suspect that an online account (Google Workspace, X, etc.) has been compromised, start with the following steps:
   
   a. Change the password immediately.
   b. Review all signed-in devices and log them all out. Almost every service that you use has this capability.
   c. Check for any 3rd party connections to your account (via the aforementioned OAuth sign-in process) and delete any 3rd party connections that you are not 100% sure are legitimate. To be safe, you can always delete all 3rd party connections and start over. Better safe than sorry.
4. If one of your social media accounts, your email account, or your Telegram account is compromised, notify all impacted parties immediately. Notify via a different, trusted means of communication. Circling back to the start of my article, it would have helped quite a bit had our employee known that a Telegram contact was compromised. Don’t put your own contacts through the same experience. A proactive call, text, or (if you must) an email can go a long way if you are concerned that Telegram is compromised.

This article could probably go on for another 10 pages, but perhaps the first and most important lesson that you should learn about operational security is that the inertia of doing nothing is your most formidable foe. Start with the top 5. If things go wrong, respond immediately and effectively as I have outlined above. Attackers are always looking for the path of least resistance. Do your best to make sure that path is not you.

What I didn’t get to in my top 5

If you hit the top 5 are you done? Not really. There is more to go in our roadmap at Certora, and I would recommend the same for any company:

• Enforcement is important. The goal here is to make sure that sign-in to critical services is gated by basic checks on device posture: recent OS patches are installed; EDR is active; etc. Google offers Context-Aware Access. Microsoft has Conditional Access. Pick one and implement it.
• SSO across all 3rd party apps you use online is essential. It is often the case that most, but not all, accounts that we and our customers use are protected by strong MFA methods. Those few that are unprotected will be compromised. Enforce uniform sign-in security standards with SSO. For Google Workspace users, this is often as simple as clicking the “Sign-in with Google” button. It couldn’t be any easier, so stop signing in with your one-off, MFA-less account ASAP.
• If you have to use an Authenticator app for TOTP or you install an app for your password manager, take the extra step to enable biometric authentication when you open that app. Google Authenticator has this feature, as does Microsoft Authenticator and apps from 1Password and BitWarden.
• Protect your password manager browser extension. Require biometrics to unlock it, and require master password re-entry once every 24 hours. Protect root account credentials by forcing master password re-entry to view them.
• Most of our work these days is done in a browser. Malicious browser extensions are a real and present danger (especially given the recent explosion of supply chain attacks against the Javascript/Typescript ecosystem). Chrome Enterprise is one good way to implement organization-wide policies for plugin control.
• Built-in phishing protections from Google Workspace and Microsoft M365 could use some improvement. There are great 3rd party tools out there to help protect your users against the latest phishing attacks. Adopt one.
• Users need help to stay safe. Phishing training is not expensive, and it is effective. There are many good solutions on the market available for phishing training. We happen to use KnowBe4, but other good options are out there.
• Users access email on their mobile devices all the time, but not all email clients are created equal. In particular, the gmail app displays phishing warnings on emails that Apple Mail does not. Adopt the gmail app if you are a Google Workspace customer (or the Outlook app if you are a Microsoft M365 customer).
  
  If you made it this far, the next round of OpSec is beyond the scope of this article. However, you are not done yet… either ask me directly what comes next, or perhaps that will be the topic of a future article.