Jump to content

Welcome to the Freedomain Radio Message Board


Sign In 

Create Account
If you're interested in joining the philosophical discussion, click "sign in" or "create account" on the right of the page. If you're creating a new account, please be sure to include an explanation as to why you're interested in joining the message board community. This verification requirement is included to cut down on possible troll and spam accounts.

If you have supported Freedomain Radio financially and would like immediate access to the message board - or - your donation status is incorrect, please contact Michael at operations@freedomainradio.com with your information and the situation will be addresses ASAP.
 
Guest Message by DevFuse

Saturday April 12th, 2014: Toronto Bitcoin Expo

Sunday April 13th, 2014: Sunday Call In Show at 10am Eastern

Monday April 14th, 2014: Peter Schiff Radio Show at 10am Eastern

Wednesday April 16th, 2014: Wednesday Call In Show at 8pm Eastern


56 Philosopher King files - 71 Gold files - 39 Silver files - 50 Bronze files


Update: A new silver level file on Peer Influence and Envy has been added! Click here to donate if you'd like access to the various premium sections. If your donator status is incorrect, please contact Michael at operations@freedomainradio.com with the relevant information and it will be corrected as soon as possible.

Photo

Suggestion: CDN to speed-up downloads and reduce ping

cdn podcast download

  • Please log in to reply
29 replies to this topic

#1
Fel1x

Fel1x

  • 18 posts
Hi
I'm experiencing really slow download speeds lately, 300Kb/s on average. Usually my speeds have been around 1.7Mb/s. My ping is avg. 225ms on 4G LTE and I'm 27 hops away and I live in Finland. Have you considered a CDN for delivering podcasts?
  • 0

#2
JamesP

JamesP


  • 3637 posts

We were getting totally slammed last night and the server was barely accessible. We put a rate limit on the downloads so that people are still able to access them, but that unfortunately means that people will be slowed down.

 

We are looking into other alternatives.  If you have any specific cost-effective suggestions, I would be happy to hear them.


  • 0

Connect with me: @jamesapyrich, Facebook, james-a-pyrich on Skype


Meetup with fellow local Freedomainers!

 

I accept BTC: 1DGcCf52Tnyc6pJbyQUwbiLj6Pkt6qHa3Y


#3
Wesley

Wesley

    Self-Excavator


  • 1222 posts

For starters, I will put a link to the torrent thread for any large number of downloads or more regular users:

http://board.freedom...ent#entry317261

 

Besides that, I think we may need information as to bandwidth usage and what cost is considered low cost. Doing a quick Google search yields sites with various tiered plans:

http://www.maxcdn.com/pricing/ (Monthly, click button for yearly)

https://www.cloudflare.com/business (bottom of page, start at $200, though they likely get a bit higher with higher requirements)

http://aws.amazon.com/cloudfront/ (Only pay for number of requests and data transferred)

 

This is just from searching CDN quick on Google, so it is easily possible that none of these are good options or that there are better options than CDNs available.


  • 0

#4
Fel1x

Fel1x

  • 18 posts

I found that CacheFly has ready integration for podcasts, excellent reviews and lots of PoPs. It just came to my mind that a CDN could also be for donators only for added value.

http://www.cachefly.com/podcasts.html

http://small-busines...tenreviews.com/


Edited by Fel1x, 24 September 2013 - 02:36 PM.

  • 0

#5
Wesley

Wesley

    Self-Excavator


  • 1222 posts

I found that CacheFly has ready integration for podcasts, excellent reviews and lots of PoPs. It just came to my mind that a CDN could also be for donators.

http://www.cachefly.com/podcasts.html

http://small-busines...tenreviews.com/

http://www.cachefly.com/plans.html

Plan pricing page for comparison purposes.


  • 0

#6
JamesP

JamesP


  • 3637 posts
Thanks for the links. Can you make any recommendations per your experience? I'm pretty new to CDNs from the tech side of things.

I can get more information regarding the throughput expectations later. But just for now, the traffic to the site had increased such that the 100mbps network interface was completely slammed. There are no available upgrades with the current hosting provider.

The other thing to keep in mind is that we do have a bandwidth allowance of 20TB per month on the webserver so anything that allows us to take advantage of that would be a plus, but it's not absolutely required.
We also have plenty of application capacity; it's the network that limits us right now.
  • 0

Connect with me: @jamesapyrich, Facebook, james-a-pyrich on Skype


Meetup with fellow local Freedomainers!

 

I accept BTC: 1DGcCf52Tnyc6pJbyQUwbiLj6Pkt6qHa3Y


#7
Kevin Beal

Kevin Beal

    :)


  • 1398 posts

We use cloudfront and Amazon S3 at the company I work at. We don't get 20TB / month of bandwidth by any means, but we chose that solution precisely because it is so scalable. And it's an awesome service using cloudfront because it's in the cloud, cached, quick etc.
 
Their S3 pricing table looks like this:

(Standard Storage  vs  Reduced Redundancy Storage  vs  Glacier Storage)

First 1 TB / month
$0.095 per GB
$0.076 per GB
$0.010 per GB

Next 49 TB / month
$0.080 per GB
$0.064 per GB
$0.010 per GB

Next 450 TB / month
$0.070 per GB
$0.056 per GB
$0.010 per GB

Next 500 TB / month
$0.065 per GB
$0.052 per GB
$0.010 per GB

Next 4000 TB / month
$0.060 per GB
$0.048 per GB
$0.010 per GB

Over 5000 TB / month
$0.055 per GB
$0.037 per GB
$0.010 per GB
 
Cloudfront's pricing is very similar I believe


  • 0

"There is no law, no compulsion, no law of physics or man that is preventing you from living the life that you want" - Stef (The Greatest Gift in the Entire Universe)


#8
Wesley

Wesley

    Self-Excavator


  • 1222 posts

Maybe I am confused, but wouldn't using a Content Delivery Network have the content on multiple servers which would each have their own network interface speed? Thus, it would be something like 100 mbps for each distinct server. Maybe I am completely misunderstanding.

 

Short Overview of CDN costs/benefits:

http://whatismyipaddress.com/cdn

 

Many of the older podcasts may not need the boost in download speed, or the torrents could be encouraged. However copying the newest/ most trafficked podcasts to many servers to download from would improve latency and would make it so you wouldn't run into the network speed cap.

 

If someone has better knowledge to weigh in on this, please do.


  • 0

#9
Kevin Beal

Kevin Beal

    :)


  • 1398 posts

Maybe I am confused, but wouldn't using a Content Delivery Network have the content on multiple servers which would each have their own network interface speed? Thus, it would be something like 100 mbps for each distinct server. Maybe I am completely misunderstanding.

 

Short Overview of CDN costs/benefits:

http://whatismyipaddress.com/cdn

 

Many of the older podcasts may not need the boost in download speed, or the torrents could be encouraged. However copying the newest/ most trafficked podcasts to many servers to download from would improve latency and would make it so you wouldn't run into the network speed cap.

 

If someone has better knowledge to weigh in on this, please do.

 

You also benefit from getting data from a closer server farm. If one server goes down, it's still up and available. They often cache that kind of thing so it's in memory and don't have to read from disk. A whole lot of benefits that I've never heard about, I'm sure.

 

A lot of people use CDN's when they want to reference CSS and Javascript libraries that lots of websites have (Bootstrap or jQuery for example) so that it's cached in the browser the first time you visit a website that requires that resource, and you don't have to download it any of the next times you visit a page that uses it.

 

CDN's are all about uptime so that when you separate your app server from a huge list of files on a CDN, your app server can go down and all your files are still up.


Also, they are built to scale so that you don't have to worry about huge numbers of visitors (except in how it affects your wallet).


  • 0

"There is no law, no compulsion, no law of physics or man that is preventing you from living the life that you want" - Stef (The Greatest Gift in the Entire Universe)


#10
JamesP

JamesP


  • 3637 posts

So yesterday we had 500GB of bandwidth consumption. That would include us getting swamped and having to rate limit during the last few hours of the evening.  I can only suspect that today would be similar.

 

We don't have 500GB days every day, but it's reasonable to expect that we will be seeing more of them in the future.  We're averaging around 250GB per day right now.

 

Yay growth! :)


  • 0

Connect with me: @jamesapyrich, Facebook, james-a-pyrich on Skype


Meetup with fellow local Freedomainers!

 

I accept BTC: 1DGcCf52Tnyc6pJbyQUwbiLj6Pkt6qHa3Y


#11
Fel1x

Fel1x

  • 18 posts
Here's a good overview of why latency matters and why a CDN is a good idea. The more time you spend transmitting to client the longer the queue gets. Link to original text.
http://blog.cachefly...e/#.UkKHFqdRWId

Queuing Delay

To understand queuing delays, first you must understand the nature of interface utilization. For
example, a 1GE port may be said to be “50% utilized” when doing 500Mbps, but what this actually
means is 50% utilized over some period of time (for example, over 1 second). At any given instant,
an interface can only be either transmitting (100% utilized), or not transmitting (0% utilized).
When a packet is routed to an interface but that interface is currently in use, the packet must be
queued. Queuing is a natural function of all routers, and normally contributes very little latency to
the overall forwarding process. But as an interface approaches the point of saturation, the
percentage of packets which must be queued for significant periods of time increases exponentially.
The amount of queuing delay that can be caused by a congested interface depends on the type of
router. A large carrier-class router typically has a significant amount of packet buffering, and can add
many hundreds or thousands of milliseconds of latency when routing over a congested interface. In
comparison, many enterprise-class devices (such as Layer 3 switches) typically have very small
buffers, and may simply drop packets when congested, without ever causing a significant increase inlatency.

Matt: Once you’ve read this once or twice, it makes all the sense in the world.. But did you ever really look at a gige port doing 512kbps and think of it as operating at 100% utilization?


  • 0

#12
shirgall

shirgall

  • 40 posts

I've been trying to download the latest episodes for about two days now. :(


  • 0

#13
JamesP

JamesP


  • 3637 posts
I appreciate the article. I wasn't trying to say that a CDN is a bad idea but what we need doesn't come cheap. I'm not on the financial side here, though, and we are getting quotes, so we will have something in place before too long.
  • 1

Connect with me: @jamesapyrich, Facebook, james-a-pyrich on Skype


Meetup with fellow local Freedomainers!

 

I accept BTC: 1DGcCf52Tnyc6pJbyQUwbiLj6Pkt6qHa3Y


#14
MMD

MMD


  • 754 posts

Howdy everybody, thanks for the feedback and ideas about how to solve this problem.  We're going to invest in a $600/month CDN package to solve this issue and allow us to scale properly for any future growth.

 

It's a significant monthly bill, but it's an important investment that is unavoidable at this point. Apparently we did 750 GB of downloads in the last 24 hours, which is absolutely insane.

 

We should hopefully have everything set up in the next few days. Fingers crossed.

 

If anybody would like to chip in to help offset some of these costs, that would certainly be appreciated.

 

http://www.freedomai...com/donate.aspx


  • 1

#15
Fel1x

Fel1x

  • 18 posts
Whoa, that's great! Congratulations to Stef and all the people who have helped him to achieve this milestone!
  • 0

#16
Filosophize

Filosophize

  • 32 posts

Why aren't torrents of recent/new podcasts posted?

 

I only see torrents for old content and even those don't seem to be working anyway...


  • 0

#17
JamesP

JamesP


  • 3637 posts

If you've been trying to download the recent podcasts (anything over 2400), you might want to restart your downloads... the CDN will treat you so kindly and gently...


  • 1

Connect with me: @jamesapyrich, Facebook, james-a-pyrich on Skype


Meetup with fellow local Freedomainers!

 

I accept BTC: 1DGcCf52Tnyc6pJbyQUwbiLj6Pkt6qHa3Y


#18
Fel1x

Fel1x

  • 18 posts
I'm downloading fdr2473 and I'm averaging about 100Kb/s. I checked my connection logs and looks like I connected to godaddy instead of the cdn.

Edited by Fel1x, 27 September 2013 - 08:52 AM.

  • 0

#19
Reason

Reason

  • 39 posts

If you've been trying to download the recent podcasts (anything over 2400), you might want to restart your downloads... the CDN will treat you so kindly and gently...

Sweet, I can finally download the sunday and Wednesday shows.


  • 0

#20
JamesP

JamesP


  • 3637 posts

I'm downloading fdr2473 and I'm averaging about 100Kb/s. I checked my connection logs and looks like I connected to godaddy instead of the cdn.


When did you start downloading 2473?
  • 0

Connect with me: @jamesapyrich, Facebook, james-a-pyrich on Skype


Meetup with fellow local Freedomainers!

 

I accept BTC: 1DGcCf52Tnyc6pJbyQUwbiLj6Pkt6qHa3Y


#21
Fel1x

Fel1x

  • 18 posts
Right before I posted my message and it still happens.
  • 0

#22
Fel1x

Fel1x

  • 18 posts
For some reason 2473 downloads from go daddy, but others download from London. My ping from the cdn is 62ms, when from go daddy it was 225ms.

Edited by Fel1x, 27 September 2013 - 07:38 PM.

  • 0

#23
JamesP

JamesP


  • 3637 posts

OK, I figured out what happened, sorry about that! That file was omitted in the upload because the filename didn't match the pattern I was using.  It should be available in about an hour.


  • 0

Connect with me: @jamesapyrich, Facebook, james-a-pyrich on Skype


Meetup with fellow local Freedomainers!

 

I accept BTC: 1DGcCf52Tnyc6pJbyQUwbiLj6Pkt6qHa3Y


#24
Fel1x

Fel1x

  • 18 posts
The cdn is working quite nicely, I'm getting download speeds of around 4,8MB/s.
  • 0

#25
Wesley

Wesley

    Self-Excavator


  • 1222 posts

Beautiful guys. Quick turn-around to find a solution to the "problem" of growth. Great stuff!


  • 0

#26
Fel1x

Fel1x

  • 18 posts
I was just wondering that which cdn provider are you using?
  • 0

#27
fer

fer

  • 85 posts

OK, I figured out what happened, sorry about that! That file was omitted in the upload because the filename didn't match the pattern I was using.  It should be available in about an hour.

http://media.freedomainradio.com/feed//handbook_human_ownership/FDR_Book_the_handbook_human_ownership.mp3

and

http://media.freedomainradio.com/feed//against_the_gods/FDR_Book_against_the_gods.mp3

are downloading really slowly, heads up if they aren't on the cdn yet.


Edited by fer, 05 October 2013 - 12:13 AM.

  • 0

#28
Kevin Beal

Kevin Beal

    :)


  • 1398 posts
http://media.freedomainradio.com/feed//against_the_gods/FDR_Book_against_the_gods.mp3

The url shouldn't look like that...

 

It's got a double forward slash after feed, also it's missing the .cdn subdomain.

 

This may be a little faster:

http://cdn.media.freedomainradio.com/feed/against_the_gods/FDR_Book_against_the_gods.mp3

  • 0

"There is no law, no compulsion, no law of physics or man that is preventing you from living the life that you want" - Stef (The Greatest Gift in the Entire Universe)


#29
fer

fer

  • 85 posts

Ohh I see, books are on the cdn but the fdrurl shorteners are still pointing outside the cdn:

http://freedomainrad.../FreeBooks.aspx


  • 0

#30
FriendlyHacker

FriendlyHacker

  • 245 posts

Some pretty good optimization tips here: http://www.cnpintegr...ite-performance

 

Not actually Joomla only related, those tips work on any site. Written by some kind of optimization expert (me).


  • 0





Also tagged with one or more of these keywords: cdn, podcast, download

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users