Scalability on index

Jul 15, 2016 at 6:52 AM
Hi there,

I have around 200,000 registered users, using version 1.5.4 and it is running quite slow. I noticed that emails are not indexed, when I found that KeyHelper.GeneratePartitionKeyIndexByEmail is deprecated due to "User only for 1.2.9.2 and lower. PartitionKey for the email index has changed to the userid to support non-unique email addresses".

I don't use non-unique email addresses, and are not going to. Any chance I can migrate back to 1.2.9.2, so I can get up to speed with the indexing. I guess I just have to build a script that recreates all the indexes, but are there anything else I should be aware of?

And do you recommend downgrading for scalability or should I rather look into upgrading to version 2?

Best regards and thank you for the lovely work,

Søren
Coordinator
Jul 25, 2016 at 4:27 AM
Hi @madebysoren:

Yes, I see exactly what you are talking about. I will get a plan together to address the problem going forward in a new release. I would rather fix the issue than have you roll back to previous version. Taking that index out was not a good decision for sure. I will try to get a fix together soon. Sorry for the trouble.
-Dave
Jul 25, 2016 at 9:45 AM
Hi again Dave,
Thank you for the answer and the reassurance that this is being worked into the next version. However we cannot wait for that version, so we are going to solve this now. I agree with you, that it's a bad idea to roll back anything. However I think that we need to "reimplement" the email indexing.
So we are looking at a possible hotfix where we add the indexing back on for emails, and create a script to loop over all of our existing users, to create an email index for each of them. Are there any good ways to extend and overwrite those specific classes/methods needed for this operation?
Best regards and thank you for your work,
Søren
Coordinator
Jul 26, 2016 at 4:17 PM
Hi Søren:

Do you happen to use the email address as the username too? or could the username be different from the email address? If the email address is used for the username that would be the best case for a hotfix and no data change would be needed. You could make the below hotfix change and it should work.
private TableQuery FindByEmailQuery(string plainEmail)   
           {               
                return GetUserIdByIndex(KeyHelper.GenerateRowKeyUserName(plainEmail), KeyHelper.GenerateRowKeyUserEmail(plainEmail));   
            }  
The email index was changed in 1.5 so a data fix would be needed to rollback to 1.2.9.2.

Thanks,
Dave
Jul 26, 2016 at 4:24 PM
Hi Dave,
Thank you for your effort. We really appreciate it!
We are not using emails as usernames; we allow Facebook login, email login and anonymous login (which basically means that we assign an available generated username).
So we need index for FB (works and is fast), username (works and is fast) and email (there is none).
My suggestion would be to re-add the KeyHelper.GeneratePartitionKeyEmail and KeyHelper.GenerateRowKeyEmail for the time being and hook them up accordingly. We've planned to work on this Thursday/Friday. If you come up with a brilliant solution before that, we'll just be clapping our hands. But I definitely think it makes sense to index both usernames and emails for fast retrieval.
Best regards,
Søren
Jul 29, 2016 at 6:13 AM
Hi Dave,

Any news? We are starting our hotfix works today, so please let me know if you've already onto something.

I'll of course update you with a potential fix.

Best regards,

Søren
Jul 29, 2016 at 6:43 AM
It seems like the download link is broken; can you help me out here?

http://download-codeplex.sec.s-msft.com/Download/SourceControlFileDownload.ashx?ProjectName=identityazuretable&changeSetId=33729

Is there anywhere else I can fetch the code to stitch up a fix?

Best regards,

Søren
Jul 29, 2016 at 10:10 AM
Hi again,

I've managed to get the link working after a couple of tries. I've separated the email and username indexes, so they work separately. Meaning that we have quick access whether using an external login provider, username or email. It seems to work quite well. Can you take a look at it, and let me know if there are any pitfalls or warnings that I should be aware of?

https://github.com/madebysoren/aspidentity-azurestorage/commit/9d69ab8ce13f819df2b68c4abe9ac017a6c30063

Looking forward to hear from you,

Best regards,

Søren
Coordinator
Sep 6, 2016 at 4:03 PM
@madebysoren: The index issue has been fixed in release 1.6 (today) and I also wrote a migration utility to fix the existing data for the email index. https://github.com/dlmelendez/identityazuretable/releases/tag/v1.6
Marked as answer by dotnetdavy on 9/6/2016 at 9:03 AM
Sep 7, 2016 at 5:57 AM
Hi @dotnetdavy,

Looks good. There's a little change from our implementation I believe. Our fixes has been running smooth for a month or so, but will check out your upgrade at an appropriate time. Thank you for the attention to the issue.

Best regards,

Søren