What happened here? How did we find this? And..

what is rent anyway?

In Solana, we have this thing called "rent". Basically, when you want to store some state on the blockchain, you have to create an "account" to store it, and you pay rent because storage isn't free. There's been a bunch of changes over the years to how rent works which aren't really relevant here except for SIMD-194.

The Solana runtime stores the current rent parameters in the Rent "sysvar". ^[1]

simd-194

Before SIMD-194's activation, the Rent sysvar looked like this:

pub struct Rent {
    pub lamports_per_byte_year: u64,
    pub exemption_threshold: f64,
    pub burn_percent: u8,
}

SIMD-194 changes the rent semantics by:

• Deprecating the use of the exemption_threshold field
• Halving exemption_threshold from 2.0 to 1.0
• Renaming lamports_per_byte_year to lamports_per_byte
• Doubling lamports_per_byte from 3,480 to 6,960

The stated goal of 194 is to replace all f64 math for rent calculations in official SDKs with u64 math, which reduces the compute unit consumption on functions like Rent::minimum_balance from 256 CUs to 8. ^[2] Given that SIMD-84 disabled rent collection years ago, this change makes a lot of sense.

194 activated on March 17th on Solana mainnet.

// Post SIMD-194 rent layout
pub struct Rent {
    pub lamports_per_byte: u64,
    #[deprecated(since = "2.X.X", note = "Use only `lamports_per_byte`
instead")]
    pub exemption_threshold: f64,
    pub burn_percent: u8,
}

program frameworks & test harnesses

If you're writing a Solana program, you're probably using a framework of some type. These abstract away things like manually deserializing accounts & instruction data from the input data pointer which usually don't really matter. The important thing here is that they also usually provide wrappers for dealing with sysvars, since they're standardized across the cluster. You call Rent::get(), it makes the sol_get_sysvar syscall, and voilà: you have a Rent that you can do some stuff with.

The value of the sysvar returned by sol_get_sysvar is set by the program's runtime environment. In a test environment, the sysvars are set by the test harness when it constructs the runtime that your programs will be simulated against. There's no builtin mechanism for making sure that the value set for a given sysvar is realistic or meaningful at all, so you (or the test harness) can set them to whatever you want.

CreateAccount::with_minimum_balance

Pinocchio is a very popular framework for writing performant Solana programs.

Pinocchio's CreateAccount::with_minimum_balance is a nice helper for creating a new account of a given size & funding it with the minimum amount of lamports required for rent exemption. This function basically invokes the system program to initialize a new account, and importantly, ask the Rent sysvar what the "minimum balance for rent exemption" is for an account of this size.

The way it asks that is Rent::try_minimum_balance, which calls Rent::minimum_balance_unchecked.

// pinocchio 0.11.1
#[inline(always)]
pub fn minimum_balance_unchecked(&self, data_len: usize) -> u64 {
    (ACCOUNT_STORAGE_OVERHEAD /* 128 */ + data_len as u64) * self.lamports_per_byte
}

The implicit assumption that's made here is that the exemption_threshold on the Rent sysvar is 1.0. Someone took the easy way out of avoiding floating-point math in program SDKs and didn't bother to check what the exemption_threshold actually was before deciding how many lamports to fund this account with.

sysvar mismatches

So what happens if the test harness provides the pre-194 rent layout with lamports_per_byte_year as 3,480 and exemption_threshold as 2.0 to the runtime, and we try to create a token account, which is 165 bytes?

Pinocchio's Rent::try_minimum_balance will return (128 + 165) * 3_480, so the new account will be funded with 1,019,640 lamports.

The token program's handler for the InitializeAccount3 instruction checks whether or not the to-be-initialized account is rent-exempt, that is, its lamports balance meets or exceeds the minimum cost for this account to stay around. This value is determined by the Rent sysvar's parameters. To figure this out, it calls solana_rent::Rent::is_exempt, which similarly to pinocchio::sysvar::Rent ^[3] calls its implementation of minimum_balance_unchecked.

What does solana_rent::Rent::minimum_balance_unchecked do?

#[inline(always)]
pub fn minimum_balance_unchecked(&self, data_len: usize) -> u64 {
    let bytes = data_len as u64;

    // There are two cases where it is possible to avoid floating-point
    // operations:
    //
    //   1)  exemption threshold is `1.0` (the SIMD-0194 default)
    //   2)  exemption threshold is `2.0` (the current default)
    //
    // In all other cases, perform the full calculation using floating-point
    // operations. Note that on BPF targets, floating-point operations are
    // not supported, so panic in that case.
    #[allow(deprecated)]
    if self.exemption_threshold == SIMD0194_EXEMPTION_THRESHOLD {
        (ACCOUNT_STORAGE_OVERHEAD + bytes) * self.lamports_per_byte
    } else if self.exemption_threshold == CURRENT_EXEMPTION_THRESHOLD {
        2 * (ACCOUNT_STORAGE_OVERHEAD + bytes) * self.lamports_per_byte
    } else {
        #[cfg(not(target_arch = "bpf"))]
        {
            (((ACCOUNT_STORAGE_OVERHEAD + bytes) * self.lamports_per_byte) as f64
                * f64::from_le_bytes(self.exemption_threshold)) as u64
        }
        #[cfg(target_arch = "bpf")]
        panic!("Floating-point operations are not supported on BPF targets");
    }
}

Oh! This is clearly not what our pinocchio calculation did. Whoever wrote this code decided wisely against assuming that all consumers forever will use the post-194 layout. You can see that this handler is also perfectly capable of processing any arbitrary f64 value for exemption_threshold. This is certainly a better design, even if everyone is supposed to be using the post-194 layout.

Unfortunately, the implementation drift in minimum_balance_unchecked causes a bit of a problem for our test environment.

Specifically, if the provided Rent sysvar's exemption threshold is CURRENT_EXEMPTION_THRESHOLD (1.0), the calculation here will be 2 * (128 + 165) * 3_480. The savvy reader may notice that this is twice as much as we initialized it to, and so our transaction will fail with TokenError::NotRentExempt.

Fundamentally, the problem here is that while pinocchio's interface assumed the post-194 rent layout, the test harness initialized the runtime environment with the pre-194 rent layout. The token program didn't care either way, as long as the account was rent-exempt. ^[4]

There's a couple obvious ways to fix this:

• Have CreateAccount::with_minimum_balance handle both rent layouts the same way solana_rent::Rent does.
• Make the test harness provide the post-194 rent layout to adapt for upcoming mainnet changes.

So how was this actually fixed?

program frameworks & test harnesses, part 2

When I found this, I was using pinocchio-system v0.6.0. Sometime between v0.5.0 and v0.6.0, this PR was merged, which switched the minimum_balance_unchecked semantics to assume a post-194 layout. Funny enough, v0.5.0's minimum_balance_unchecked actually did handle both layouts correctly.

Either way, this change generally shouldn't have been a problem. pinocchio-system v0.6.0 released on April 8th, 22 days after 194 activated on mainnet. One third-party harness, LiteSVM, adapted to the new layout in v0.11.0 3 days post-activation.

Pinocchio is primarily maintained by Anza. Anza is the core dev team behind the Agave validator client, and among other things, the Mollusk test framework.

Mollusk did not switch rent layout until May 26th. This means that if you upgraded to pinocchio-system v0.6.0 anytime before May 26th and used Mollusk to drive your program test suite, you probably saw some bizarre InsufficientFundsForRent errors caused by this bug.

conclusion

Weird little bugs like this are not uncommon in Solana. They probably happen at a much higher rate than in other types of programming. This is extremely frustrating devex. We've just seen that using two very popular crates, maintained by the same team, could have forced you into fighting a very sneaky implementation drift bug in an oft-forgotten part of the protocol.

Solana devs typically call this "chewing glass".

I don't really enjoy the taste of glass. It's very similar to blood.

example repo

If you want to see this bug in action, this repo shows what happens on pinocchio-system v0.6.0 when simulated against a couple different versions of LiteSVM & Mollusk.