dmnt3hi, ive got about 5 gigs worth of json files each about 500 kb big that im trying to insert into a mongodb using pymongo. this is take 81 days which is obviously to slow. Is there any way to speed this up ?
joannacdmnt3: That means on average one 500kb file takes 12 mins to insert?
joannacThat is incredibly slow. Where are you getting these numbers from?
dmnt3sorry i got that number from another guy im working in team, it seems its more like 14 hours which still seems like a long time
SpiXyHi all, I have a question regarding how simultaneous heavy insert/update and reads work. Consider I'm doing an aggregation on several million documents, while the exact same documents are constantly (1. updated, 2. new documents meeting the conditions of the update are inserted).
SpiXyhow is mongodb gonna handle this ? Do i need to use a locking mechanism to disable updates and writes when reading the huge chunk ?
SpiXyor is it safe to just read and let mongodb handle it ? (will the read take much longer since the things that it has to read are increasing/changing by the second)
SpiXywill it return consistent results ? since maybe at the start of the read, some documents matched the condition for update, but after a minute (which the read takes longer than usually, there is no guaranty that the same document now meets the condition, since it might have been updated
SpiXyThanks in advance for any help ! :)
Dussedhey Derick - wondering if you could assist on an issue we're having. Getting weird dup key errors on an _id_ index - however, the entries seem to be completely different, and one of them doesn't even have an _id, so no idea how it's colliding on _id_ index
Derickhow long are the values in _id ?
Derickjust normal object ids?
Dussedna we moved over to our own guuids
Dussedwell, kinda
Dussedd6df7ada-19e6-fa34-ce07-1d8d4cfcb373-6b4d16ad939c-4166822-1509445083
Dussedbeen trying many many things, including new collections, dropping all indexes, sparse indexes, nothing seems to resolve this
Dussedalso noted a trend - 99.9% of the errors come from our asia nodes - the server is in frankfurt, it was moved to improve lantency
Dussedeach request to DB from asia node takes ~170ms
hillctGood morning all. I’m upgrading from MongoDB 2.4 to 3.4, following the stepped, major release intermediate path. Do I need to upgrade the Storage Engine to WiredTiger in the 3.0 upgrade step, or can I export the MMAPv1 data and reimport once, using 3.4 (which still seems to support MMAPv1) ?
threespadesyou are not required, and you can dump and import the data https://docs.mongodb.com/manual/tutorial/change-standalone-wiredtiger/
sandman13You can save a lot of space if you use WiredTiger as storage engine
hillct@threespades @sandman13 Thank you. I do intend to upgrade the storage engine, just trying to sort out the optimal seteps in my rather long upgrade path (for my dev enviroment). Also, since I’ve not deployed a significant Mongo project since v2.4, Can you advise me as to the best (local deployment) web based or Mac desktop Mongo admin client tool?
threespadescompass for official tools, robomongo for third party
hillct@threespades Thanks
sandman13BTW does robomongo support ~/.ssh/config ?
Derickdo you mean normal SSH tunnels, or?
sandman13No
Derickthen what do you mean?
sandman13We have central server to SSH into machines
Dericklike an ssh proxy server or something?
sandman13Yeah
Derickhm
threespadeshttps://github.com/Studio3T/robomongo/issues/1089 remains unanswered
Dericki can't really help here :)
sandman13Derick: https://bpaste.net/show/d74548806a84 setup is something like that
Derickright, we use that for accessing php.net servers too
sandman13Why open extra port when you can use preconfigured ~/.ssh/config? :)
hillct@threespades I guess they renamed robomongo, Studio 3T. Very impressive. Previously I was using MongoHub on the desktop, and RockMongo for a web UI. Both were nightmares
threespadesrobo 3t is different from studio 3t i believe
hillctLooking over their intro materials now. Seems to be a bit more robust, not quite so light…
hillctfrom the same devs though
threespadesah guess they are the same people
sandman13Yeah
sandman13Robo 3T is just Robomongo rebranded
sandman13IIRC
hillctI see. The merged the projects RoboMongo and MongoChef, under the same umprella, but maintain them as different projects and distributions
hillctWill check out both, I guess…
hillctumprella… wow, I love those typos where you can’t get away with justifying it by saying the P key is next to the B key…
hillctThe storage utilization improvement with WiredTiger is AMAZING
Derickyup :D
loquacehello
kurushiyama@loquace Hello!
loquace;)