![]() ![]() Cloud backup contains many kinds of data such as files, folders, even applications and systems. Its main purpose is to protect data from disasters and recover data after loss. So, what is cloud backup, and what kind of cloud backup software do we need?Ĭloud backup is a way to save your data on the cloud. Because cloud backup is the development trend of backup, which is going to make up for the shortcomings of traditional backup. I would assume this is what's happening and if you can corroborate it in your logs per above then the best thing to do is create a new policy for Backblaze using ISDB or manually specifying Backblaze IP ranges and setting the TTL ( set session-ttl ) for sessions using this policy so something like 72 hours or longer possibly.In such a digital age, we are more often to talk about cloud backup. But from FortiGate perspective it's not a current connection as its been timed out so you get the errors and RSTs from the FortiGate because the FortiGate wants TCP handshake now. If so, from Backblaze's persepctive this IP is still serviceable for backup communications and it skips the TCP setup assuming your client still has the session opened (no one send a FIN or RST packet. You should also be able to go back to your historical traffic logs and see if there was a previous session opened to the same IP for this backup session. You should be able to confirm by seeing which Backblaze IPs are causing the " no session matched" logs. If Backblaze is setting up new parallel sessions (sounds like it given your frequency of session rotation count) to service the backups, then is it possible it is going back to old sessions that it was previously using but they are now timed out and no longer in the FGT session table? Do your logs show this? Minimum chunk size (configured) = 60MB (but we've tried 10,20,64,100,120,250)Īny ideas on what to try next would be appreciated! If you need more info, just let me know.Frequency of session rotation (observed) = 1 new session every 5-6 minutes.Maximum number of concurrent connections (configured) = 64 (but we've tried 8,12,16,24,32).Max number of concurrent sessions (observed) = 68. ![]() ![]() SSL policy = no-inspection (the factory policy).Destination IPs = Multiple (but we have all of them in an address group).Also, we have the TCP timeout on this policy set to 2 hours. There is hardly any other traffic on this network and the CPU never even reaches 10%. The firewall is a 61F, which we know has an inspection limit of 700Mbps, but we have the backup capped at 500Mbps, and we've created a policy rule that exempts these jobs from inspection altogether (Fortinet verified the policy was created correctly and is correctly matching the traffic). We do have a ticket open with Fortinet but, TBH, I think they're running out of ideas because it's gotten to the point where they're not even reading the notes/updates anymore and we we're losing hope in a good resolution. We know it's the firewall because if we bypass it, then the jobs work fine (all of them). This is corroborated by the firewall logs which then contains thousands of the "no session matched" errors (presumably from the server retrying before it gives up). What's real interesting, is no matter how many connections it had open, *all* of them fail at the exact same time. ![]() Nevertheless, we were lucky enough to get a capture during one of the times when the job failed quickly, and it does show data being transferred and then suddenly stops. This also makes captures extremely difficult because even if we slice to header, it's still 16K packets per second. Sometimes a job will run for 5 minutes, and sometimes it'll run for 2 days. The biggest problem trying to troubleshoot this is that it never fails at the same place. Less than 0 bytes transferred the last 60 seconds". Even in the backup logs, we can see it was in the middle of sending the multi-part upload, then suddenly it logs the error " Operation too slow. This is very strange, because on the sniffer we can see the transfer was actively going. The firewall shows an implicit deny because of " no session matched". No matter which one we start, it always starts just fine but eventually we will begin seeing a flood of TCP retransmits, and shortly after, the job fails. The jobs are all set to manual, and never run together. The backups are broken into different jobs with the largest being ~12TB, and the smallest being ~400GB (total for all is ~40TB). We are using MSP360 to transfer to Backblaze (B2). We have a large transfer that we are sending through our FortiGate but it always fails because the sessions are getting dropped on the firewall. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |