DIA Oracle misreported GRAIL/USD price - November 22nd 2023

Snapshot Proposals
1

TL;DR

  • On November 22nd, around noon UTC, there were systematic deviations in the value of some assets due to HTX (Huobi) hack.
  • GRAIL price feed was momentarily affected by such deviations, with some trades going down to price ranges of under 600 USD / GRAIL. The actual perceived market price at the time was around 1200-1400 USD / GRAIL.
  • Three users became insolvent and had their GRAIL-backed loans liquidated, losing roughly $21K in liquidation fees.
  • DIA fixed the issue with GRAIL/USD price feed. The feed has been reporting accurate values.
  • Impacted users have reached out to the team seeking compensation.
  • We invite the community to participate in a discussion.
  • Read the DIA disclosure at the end of the post for more details.

About Silo’s DIA data feeder
All oracle providers, including Chainlink, use decentralized exchanges (DEXs) and centralized exchanges (CEXs) as data sources to report the value of crypto on-chain. DIA oracle pull data from a number of CEXs and DEXs, apply filters to prevent manipulation and ensure data integrity, and then report them to some of Silo’s isolated markets: SILO; GRAIL; JONES; rDPX. Below we break down each price feed and the data sources used to report the value of the aforementioned assets.

SILO/USD: Uniswap and Camelot.

GRAIL/USD: Sources include Camelot, TraderJoe, MEXC, and others. Note Huobi was removed following the reported incident.

rDPX/USDC: Camelot and Uniswap. Note the market runs a risk label. We advise all users to withdraw funds.

PLS/USD: Sources include Uniswap and Camelot.

JONES/USD: Sources include MEXC, Camelot, Sushi.

Y2K/USD: Note the DIA oracle is NOT maintained by the Silo team. The oracle receives data from Balancer and Uniswap.

Silo terms of use
Silo’s lending app’s terms of use stipulate that third-party services like oracles cannot be monitored or controlled and therefore fall out of the responsibility of the Silo DAO. Users bear the responsibility when they deposit in Silo’s isolated markets. The lending app discloses adequate risk messages to warn users about the danger of price oracles.

For the community to address

  • Are the three users entitled to receive compensation?
  • By compensating the users, would not the Silo DAO set a precedent that can be used against it in the future? How can the DAO be responsible for the performance of an external service where prevention and control fall outside the purview of the DAO?
  • Does the DAO have an ethical obligation to stand by its community of token holders and users?

Please reply to the thread below with your response.

DIA report below
The following is a report published by the DIA team. You can direct any questions to the DIA team by replying to the thread below:

Report:
The Silo oracle experienced a deviation from the perceived market value of GRAIL. This happened in conjecture with a hack on one of the centralized exchanges that this feed uses as a source. This exchange (HTX, formerly known as Huobi) has been delisted in the process of the mitigation.

Recap of events
This hack lead to systematic deviations in some assets from the perceived market value shortly after noon UTC on Oct 22nd 2023.
GRAIL was also affected from such deviations, with some trades going down to price ranges of under 600 USD / GRAIL. The actual perceived market price at the time was around 1200-1400 USD / GRAIL.

Filters and methodologies
Most of the deviating trades were mitigated by an interquartile range outlier filter in the price calculation layer leading to the dismissal of the trades. However, in some affected time ranges the number of trades in the overall market was quite low, so that the HTX trades were considered as possible actual trades, respectively the number of HTX trades was the majority of trades we saw and the average price was close to their reported prices. It should be noted that the other sources for GRAIL prices include DEXes that usually have lower number of trades, which can lead to situations where in the 120s window of trade evaluation all or almost all trades come from the centralized exchange source.

In most of these cases the second deviation mechanism in the oracle feeder prevented updates to the oracle: The comparison to the published price from other data providers in the Silo feeder. However, as the deviation threshold in this feeder is 20% to the perceived market price, some prices were published to the oracle that were influenced by HTX prices but still close enough to the perceived market price.

For some long-tail assets the market rates are fluid and can deviate strongly under normal market conditions. This market rate deviation mechanism only triggers in extreme cases where an actual error in the price evaluation layer leads to prices that are completely unrepresentative of the overall market. False positives arising from too tight deviation thresholds should be avoided, otherwise it could happen in genuine market conditions that changes could be prevented from being reported in a timely manner.

Immediate actions
At around 1:40 pm UTC our system shut down the HTX data source entirely identifying an unusual high number of deviations overall.
Altogether, this aberration was not unfounded (these trades on HTX existed at that time after all), but regarding the hack we think that these can be considered of lesser data quality.

Follow-up actions
In coordination with the Silo team the deviation threshold will be lowered to 7.5% for all assets except RDPX, as deviations in the range of up to 20% can frequently be observed, backed by actual DEX trades from pools with reasonable liquidity.
Historical data shows that the assets are usually within these 7.5% and do not deviate out of this window.

The live composition of available markets can always be seen in the DIA app for any asset. This is also why we link this information in any CDR so that it is always transparent which data sources are involved in an oracle price.

5 Likes
3

(I’m one of the 3 affected users)

First, thanks to the team for being willing to listen and report back to some random guy who got liquidated on his relatively small position. I know not any team in crypto is like this.

Of course, as one of the “victims” i’d like to get compensated but i know i cannot pretend it. I knew the risks and the terms of use and the responsability of the misrepored price come from the exterior.

About setting a precedent, i think every case is to be analysed based on circumstances (total affected users, total value to refound, could have the team made something to mitigate the risks, etc.) and then take decisions listening to the dao, maybe some other maesures if way more founds are involved?

" Does the DAO have an ethical obligation to stand by its community of token holders and users? "
If we talk about ethics that’s up to the team. It’s correct to refound and it’s correct to not refound.
Yes i asked to get compensated but again, as one of the users affected i cant be 100% objective here.

Edit. I still have positions in silo and if i get my GRAIL back i’ll deposit right away again lol

3 Likes
4

Disclaimer: I am one of the 3 affected users and have been a long time Silo user in multiple vaults

I’d like to start by saying that ultimately oracle risks are well known in DeFi and something that all users need to be cognizant of especially when using a lending protocol such as Silo. With that said I do believe there is at least some shared responsibility here amongst users and the team as ultimately these oracles are vetted and selected by the team.

In terms of setting a precedent it is my belief that all issues related to loss of funds or hacks should be reviewed on a case by case basis with factors such as amount to backstop playing a role. To this point though we should be thankful that this oracle issue happened on a smaller vault with only 3 users affected for small $ amounts. If this had happened to one of the larger vaults regardless of fault I believe the community reaction would be much different. With that in mind it may make sense to come up with some pre-emptive initiatives similar to what Aave has done in order to maintain enough funds (community or team) in order to be able to backstop future losses as voted on by the DAO.

Finally in regards to the question around ethical obligation to stand by it’s community, I again feel like this should factor in multiple items. I do believe the DAO should feel some obligation to stand by their community in some capacity and in this case since the amount and number of users is small it is a relatively easy signal to send to the community while maintaining optionality in the future. Especially given the fact that STIP rewards were just given out to attract new users i’d argue that utilizing those rewards to keep loyal users around is a good use of funds and signal to the community that the DAO supports them during tough times.

3 Likes
5

Why deleted sir? could you repost?

1 Like
6

Thank you sir, we appreciate sharing your perspective, and we are sad you were liquidated. We will keep the conversation going and hope we can find a solution.

7

Thank you sir. We have had a great conversation on Discord and I appreciate it. We will continue the conversation and hope we can arrive at an equitable solution.

8

(I’m not affected)

Hello ser,

I changed my mind regarding my post but I can see degen making the same points I did and I believe more eloquently also.

Mainly I agree that, since the oracles are chosen by the team, some responsibility should be taken for incidents like these.

It’s like passing the buck. Where does it end? Silo blames the Oracle and the Oracle in turn blames someone else. In the end the user loses.

The user did everything correctly and lost funds with Silo. So it’s Silos fault. If Silo then wants to take up a case with DIA then that’s fine but the Silo user has nothing to do with it.

And think like this. If the Silo team was even smarter, better, more experienced that you beautiful bastards already are, could this incident be avoided?

If the answer is yes, then you have to concede that this was indeed inside your control and were you better you could avoid it. Therefore I say, take responsibility and become better. Don’t escape by way of disclaimers written in small letters.

I think if you take pride in your work and product you don’t let people lose money in your business. You take responsibility for what happened and do better. And I really like the suggested improvement with the price deviation.

In the end it’s such a small price to pay in order to gain so much.

2 Likes
9

(I’m not affected)

Hello sirs, I’ve been using Silo for a while now and I really like the UX, plus comms from the team (I used to stick to blue chip Aave & Compound, but Silo got my attention).
I believe users losing funds at a platform shouldn’t be a “hot potato” between Silo and the Oracle provider. One of the very innovation of Silo is to allow more diversity with Oracles (not only Chainlink & Uniswap TWAP, but also other ones like DIA), so the providers should also strive to be a long-term strategic partner. Thus, I think:

  1. If DIA is a serious provider on the Oracle front, this team should reimburse the damage to the 3 affected parties - in the end it wasn’t GRAIL to collapse as an asset, rather one source. IMO it’s the job of an Oracle to detect such malfunction and protect against it.

  2. Silo team internally should keep track of the methodologies of Oracles and assess them one by one. There’s a popular Volume Weighting approach, but also other ones that might be more resilient like TWAP, Median or even Liquidity weighting. I’m not saying one is better than the other, but more attention to them should be paid.

Happy to hear your position on that, and I hope DIA team will chime in to this post too.
Thanks!

2 Likes
10

Thank you, sir. Unfortunately it is not possible for the team to keep track of oracle performance and the data filtering methodologies because such systems are not open-source. It is the reason one of the reasons we signal the oracle risk to users in the UI.
For onchain twaps like Uniswap, our analytics already signal oracle risk because it is possible to calculate an objective rating - check the ratings here (some markets are not launched in the UI such as GMD):
https://analytics.silo.finance/arbitrum-one/ratings

11

I suggest the following plan:

  • Compensating users entirely with SILO tokens.
  • Impacted users vest allotted SILO amounts for 30 days on Hedgy.
  • Compensation is not an admission of responsibility. SiloDAO maintains its position on not compensating for funds lost due to 3rd-party services malfunctioning. SiloDAO is encouraged to inform current users of lending apps of current risks and the limits of DAO’s risk coverage.

Calculations:

Next steps

  • If impacted users agree, we can proceed to snapshot voting.
1 Like
12

Doesn’t a linear vesting make more sense? Or something like 3 or 4 cliffs spread on 30 days?

13

It will be weekly unlocks.

1 Like
14

(I’m not affected)

I am firmly against compensation.

It is clear in both the documentation and the UI that users take on the risk of depositing into a silo, and a core attribute of each silo is the oracle choice. I believe this line has to be held for the protocol and DAO to maintain clarity around risk and responsibility, and not to introduce any ambiguity. The small $$$ does not factor into this.

I have been party to a handful of these situations in different protocols over the years, and the concern about setting precedent is valid. If a protocol decides to compensate users once, it becomes expected in future loss events - even if the circumstances are different / less justifiable. This can (and has) create(d) a swell of toxicity between the protocol developers, the DAO and protocol users.

For this scenario, the only way I could be convinced that compensation was the right thing to do was if the team had been either obviously negligent in the implementation of the oracle feed or deliberately misrepresented or obfuscated the nature of the risk. Neither is true in this case.

In future, I could see a role for insurance or some kind of fund, but this is obviously something that has to be thought carefully about, set up proactively and with specific conditions.

2 Likes
15

Proposal defeated:
https://snapshot.org/#/silofinance.eth/proposal/0x8a47ef948597696cff3b1483213bfaa2a9f76b805629e3ecafd2e769d04a66ee